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ABSTRACT 

During 1985-86 the School Community Education Program 
(also known as the Umbrella Program), funded by the New York State 
Legislature, provided a variety of educational and training 
experiences to some 25,871 participants, including pre-kindergarten 
children and their parents; and students, teachers and supervisors 
from kindergarten through grade 12. The program consists of 44 
different projects designed to provide innovative solutions to local 
educational and school problems. The 1985-86 evaluation indicates 
that in general the program was successful: 38 projects met their 
stated objectives, and some were highly successful. Of the six 
projects that did not reach their objectives, five set stringent 
objective criteria that may have been beyond their grasp. Those 
projects that failed to meet their stated objective should be closely 
■onitored to identify the reasons for failure. Evaluation reports for 
each project are presented in four volumes. Bach report contains a 
brief project overview, describes the evaluation methodology, 
presents the findings, and offers recommendations for improvement. 
This volume. Volume III, presents evaluation reports of the following 
staff development projects: (1) Legal Outreach Program; (2) Staff 
Development in Nriting Instruction; (3) STAR LABS; (4) Mathematics 
Improvement Program; (5) Reasoning/Thinking Skills Program; (6) Arts 
in General Education (AGE); (7) Enrichment Program K-9; (8) 
Adventures in Science; (9) Early Childhood Language and Literacy; 
(10) Discovering Abilities and Improving Achievement; (11) Sun in 
One; and (12) Mastery Learning Program. Data are presented on 15 
tables. Appendices include copies of program-developed assessment 
instruments. (BJV) 
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EVALUATION SUMMARY 



BACKGROUND 

The School Community Education Program (also known as the 
Umbrella Program) provides a variety of educational and training 
experiences to a wide range of participants, including pre- 
kindergarten children and their parents; and students, teachers, 
and supervisors at all grade levels from kindergarten through 
grade 12. The program consists of 44 different projects designed 
to provide innovative solutions to local educational and school 
problems. Ten projects provided basic skills, English, and 
computer literacy instruction; ten focused on social ard 
environmental studies; five were designed for pre-kindergarten 
children, and the remaining projects provided a variety of 
educational experiences to participants. Funds were provided by 
the New York State Legislature to support program activities. 

POPULATION SERVED 

During 1985-86, the program served some 24,290 students, the 
majority of whom were elementary school pupils. In addition, the 
program served 1,226 teachers and supervisors, 245 pre-schoolers, 
and about 110 community adults in the 32 community school 
districts and selected high schools. Each project established 
different selection criteria for program participation. 

PROGRAM OBJECTIVES 

Although program objectives were designed for each specific 
project and, therefore, were varied, most concerned increasing 
participants' competence in specific skills and abilities. 

EVALUATION METHODOLOGY 

The evaluation of the program was based on a number of data 
sources: student performance outcomes on project-developed and 
standardized tests, pupil writing samples, teacher and student 
survey questionnaires, attendance rates, number of acceptances to 
special high schools, and review of two curriculum manuals. Pre- 
program and post-program daca were compared to determine mean 
differences and, when appropriate, correlated t-tests and effect 
sizes were also computed to establish statistical significance 
and educational meaningfulness , respectively. 

FINDINGS 

The 1985-86 evaluation findings indicate that, in general, 
the School Community Education Program was successful. Thirty- 
eight projects met their stated objectives. Three staff 
development projects (Arts in General Education, Sum in One, and 

Early Childhood Language and Literacy) and two pre-kindergarten 



projects (Brooklyn College Tutorial Center and Pre-School Gifted 
and Talented) were highly juccessful. All projects providing 
instruction in mathematics, writing, English, and computer 
literacy met their project objectives. in all five pre- 
kindegarten projects, participants substantially improved their 
overall performance. 

Only six projects did not meet their evaluation objectives. 
Apart from the Help: Neighborhood Center project that needs 
extensive project modifications, the other unsuccessful projects 
set stringent objective criteria which may have been beyond the 
programs' reasonable grasp. 

RECOMMENDATIQM 

In addition to the recommendations made for each project, 
the following suggestion is made for the overall improvement of 
the School Community Education Program: 

. Closely monitor those projects which failed to meet their 
stated objectives to identify reasons for failure to 
achieve criterion for success. 
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INTRODUCTION 



In 1985-86, the New York City Public Schools received 
$2,375,000 in funding from the New York State Legiiilature to 
operate the School Community Education Program (also known as the 
Umbrella Program) . It consisted of 44 different projects 
designed to provide innovative solutions to local educational and 
school programs. 

The program provided services to ifcout 25,871 participants 
in the 32 community school districts and selected high schools. 
The majority of these participants (24,290) were elementary, 
intermediate, and high school students. In addition, 245 pre- 
school children, 1,226 teachers and supervisors, and 110 
community adults also participated in the Umbrella Program. 

Evaluation reports are presented in four volumes. Volume I 
contains evaluations for ten projects which provided reading, 
mathematics, writing, English, and computer literacy instruction 
to elementary, intermediate, and high school students. Volume II 
includes evaluations for ten projects on social, ethnic, and 
environmental studies, and instruction on communication and the 
arts. Three of these projects also provided staff development 
training. Volume III contains evaluations for 12 staff 
development projects. The remaining 12 projects, presented in 
Volume IV, provided a variety of educational experiences to 
participants. Five of these projects were designed for pre- 
kindergarten children, two were concerned with the writing of 
curricula, one provided parenting skills instruction to students 
with infants, and the other four projects were designed to 
improve attendance rates, health, opportunities to gain 
acceptance to special high schools, and to foster career 
awareness among students. 

Each report contains a brief project overview, describes the 
evaluation methodology, presents the findings, provicles 
recommendations for improvement, and includes copies of program- 
developed assessment instruments. The reports are listed in 
order of budgeted function number in the table of Contents. 
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LEGAL OUTREACH PROGRAM, 1985-86 

School Community Education Program 
Program Coordinator: Jack Isaacs 
Project Coordinator: Eliot Salow 

Prepared by: 
Office of Educational Assessment 
New York City Public Schools 
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PROJECT DESCRIPTION 

The Legal Outreach Program is designed to teach the history 
and use of legal concepts to junior high school teachers in 
Community School Districts (C.S.D.s) 5 and 16 and train them to 
integrate this knowledge into their regular classroom curricula. 
A recent interview conducted by the districts' resource personnel 
showed that many residents and students in these areas have 
become disenchanted with the legal system. To help overcome this 
negative attitude, the project seeks to make students more aware 
of the strengths and limitations of our legal system as well as 
to help them understand their own responsibilities in school and 
the community. 

In 1985-86, the first year the project was implemented, 16 
teachers from C.S.D.s 5 and 16 participated in the prograun. The 
selection of participants was based on the teachers' willingness 
to participate in after-school workshops and additional training 
activities. Project activities involved a series of training 
workshops which focused on the relationship between the legal 
system and the community. The curriculum, entitled "Law in the 
Community/' related law to the legal system, the family, the 
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schools, the neighborhood, and society. Expert consultpiics 
conducted these workshops, assisted teachers in the preparation 
of lesson plans, and provided in-class support. Additional 
activities involved both teachers and their students who visited 
court houses, organized mock trials in the classroom, and invited 
guest speakers to the schools. The New York State Legislature 
provided $23 thousand to fund the project. 

The stated objective for 1985-86 was for participating 
teachers to learn skills, knowledge, and techniques necessary to 
implement a law-reTated instructional program in the classroom. 
Their competence in these areas was to be measured by a program- 
developed test. 

EVALUATION METHODOLOGY 

The evaluation of the project focused on the analysis of 
participants' scores oi> a multiple-choice, project-developed test 
(see Appendix A). The test consists of 20 items dealing with 
legal concepts, processes and infrastructure, marriage and 
divorce, child abuse, real estate, crime, and governmental 
agencies. This test was administered at the beginning and end of 
the program. 

FINDINGS 

Complete scores were submitted for 13 teachers (see Table 
1). Participants in both districts achieved gains on the 
posttest. Overall, mean pretest score was 10.9 points (54.3 

2 



11 



63403 



TABLE 1 

Teachers* Mean Raw Scores^ on a Program-Developed Test^ 

by District 
Legal Outreach Program^ 1985-86 



Pretest Posttest 
Raw Percent Raw Percent Percent 

District N Score Correct Score Correct Gain 



C.S.D. 


5 


8 


11.3 


5c. 5% 


15.4 


77.0% 


20.5% 


C.S.D. 


16 


5 


10.4 


52.0 


16.0 


80.0 


28.0 


TOTAL 




13 


10.9 


54.3 


15.7 


78.5 


24.2 



^Perfect Raw Score = 20. 

• Participants in both districts achieved gains of over 20 
percentage points. 
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percent correct), mean posttest score was 15.7 points (78.5 
percent correct) , for a mean gain of 24.2 percent. 



CONCLUSI^' , RECOMMENDATIONS 

The evaluation findings indicate that participants incre'ied 
their knowledge of the legal system as measured by the program- 
developed test. Measuring whether the project met its objective 
remains problematic, however, because the objective was too vague 
and the testing instrument could not measure it adequately. Un- 
doubtedly, teachers broadened their knowledge of the legal system 
but it is impossible to tell with the data at hand whether they 
learned the skills and techniques to implement a law-related 
instructional program. Appropriate itens to measure class imple- 
mentation of knowledge on the legal system should be included in 
the test. Alternatively, the progrcun coordinator or the consul- 
tant could evaluate teacher performance in the classroom accor- 
ding to a specifically designed checklist. In addition, the test 
could be revised since most participants knew 50 percent or more 
of the answers at pretest. This indicates that the test might be 
too easy. Those items that most participants know at pretest 
should be eliminated. Finally, project staff should de^ ign a 
specific project objective, including quantitative criteria for 
successful prograun completion. For instance, the objective could 
state, "participants will achieve a gain of at least 25 percent 
at posttest and/or participants will show mastery of 80 percent 

4 
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of competency skills in a checklist developed to measure teacher 
performance in the classroom," 
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LEGAL OUTREACH. PROGRAM 

TEACHER TRAINING WORKSHOPS 

• Er ft-? t es t /jPps tes t 
Multiple Choice Questions 



Which of the following New York acencies provides services 
to the needy including Medicaid, day care, foster care, and 
family planning? 

a. the department of labor 

b. the department of social services 

c. the department of health 

d. the board of regents - 

The highest tribunal in New* York State is: 

a. the Appellate Division 

b. the Supreme Court 

c. the Court of Appeals 

d. the Appellate Term 

Which of the following types of cases does the Family Court 
of New York not have jurisdiction: 

a. juvenile offenses 

b* child custody and support 

c. divorce proceedings 

d. paternity matters 

The trial court at the federal level is known as 

a. the district court 

b. the court of appeals 

c. the Supreme Court 

d. the federal Court of New York 

In civil trials in the State of Nevr York, the jury is 
composed of how many individuals: 

a. six 

b. eight 

c . ten 

d. twelve 

The" term voir dire refers to: 

a. a grounds for objection by an attorney 

b. a mistrial called by a judge 

c. a process of selecting prospective jurors 

d. the decision made by the jury 
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?• Plea bargaining is 

a. a process by which a defendant negotiates his fee 

with the defense lawyer, unless he is a court appointed 
attorney . 

b. a process utilized by ^e jurors to reach a decision 
in a criminal or civil .rial, 

c. a pr-?cess by which the defense lawyer is able to ob- 
tain evidence from the prosecutor,* 

d. a process by which the opposing attorneys agree on 
a guilty plea by the defendant in exchange for a 
lesser charge, 

8. Youths aged 14 to 16 whose cases are handled in the adult 
criminal court are referred to as 

a. youthful offenders 

b. juvenile offenders 

c. juvenile delinquents 

d. youthful delinquents 

9, Which of the following statements is true for a fourteen 
year old girl who desires to marry?' 

a. She cannot marry 

b. She must wait until she is sixteen 

c. She must obtain parental consent and the permission 
of the family court. 

d. She must wait until she is sixteen and obtain parental 
consent and the consant of the court. 

10. Which of the following statements is the most accurate 
statement of the law in New York: 

a. common law marriages automatically exist if two people 
live together seven years. 

b. two people who live togehter are not obligated to 
support one another, unless they expressly agree to 
do so or "formally" marry. 

c. a common law marriage, in order to be valid, must be 
certified in a court of law. 

d. common law marriages do not exist in New York. 

11. Which of the" following is not a grounds for divorce in 
New York? 

a. abandonment for a year or more 

b. imprisonment for two consecutive years 

c. cruel and inhuman treatment 

d. adultery 



16 



3 



63403 



12. In cases where child abuse is suspected, an investigator 
can remove the child when 

a. he senses that the child has been abused. 

b. their is another relative with whom the child can live. 

c. there is imminent danger of ham. 

d. the child is unable to support him or herself. 

13. Select the correct definition; 

A neglected child is one whose health is impaired by a 
lack of (1) proper food (2) clothing (3) shelter 
(4) education (5) social outlets (6) medical care. 



14.. An annulment would not be granted for which of the fol- 
lowing reasons: 

a. non-age and proper consent 

b. physical incapacity 

c. fraud or duress 

d. temporary insanity 

15. Which of the following is not considered a public assistance 
program: (1) Medicaid (2) social security (3) food 

stamps (4) supplement security income 

a. all are public assistance programs 

b. only (1), (2), (3) 

c. only (1), (3)-, (4) 

d. only (2), (3). (4) 

16. In housing matters, the warrant of habitability refers to: 

a. the tenant's obligation to keep the house in good repair. 

b. the landlord's obligation to keep the house in good 
repair. 

c. the landlord's obligation to maintain the premises in 
safe and livable conditions 

d. the tenant's obl:.gation to maintain the premises in 
safe and livable condition • 

17. In housing matters, the right of quiet enjoyment refers to: 

a. the tenant's right to make any changes which s/he wants 
CO within the leased space. 

b. the tenant's right to utilize the rented space for 
any lawful purpose which s/he so desires. 

c. the landlord's right to receive rent payments cn time 
from his tenants. 

d. the tenant's right to use the property without being 
disturbed by the landlord or other tenants. 



a. 
b. 



d. 



c. 



all the above 

(1), (2), (3), (4) 

(1). (2), (3), (4), (5) 

(1), (2), (3), (4), (6} 
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18. Under what circumstance is a search by a policeman con- 
sidered unreasonable without a warrant? 

a. when he has reasonable cause to believe a crime has 
been committed. 

b. when he is in hot pursuit. 

c. when' he has prob€d:)le cause to believe that a crime 
has been committed. 

d. when an arrest has been made. 

19. When a policeman reasonably believes that a person is 
about to commit a crime and also believes that his life 
is in danger, he can 

a. only stop and inquire 

b. stop arl frisk 

c. stop ani search 

d. search and seize 

20. . A *, teacher or administrator can search a student 



a. only if s/he has a warr?.nt. 

b. any time s/he believes that a school rule has been 
broken . 

c. only when there is a law-enforcement official present. 

d. whenever there is reasonaible cause. 
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LEGAL OUTREACH PSOGHJCIf 

TEACHER TRAINING WORKSHOPS 

Pre^t«»C/*Pos t- ces c 
Multiple Choice Answers 



1. 


b 


2. 


c 


3. 


c 


4. 


a 


5. 


a 


6. 


c 


7. 


d 


8. 


b 


9. 


c 


10. 


b 


11. 


b 


12. 


c 


13. 


d 


14. 


d 


15. 


a 


16. 


c 


17. 


d 


18. 


a 


19. 


b 


20. 


d 
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STAFF DEVELOPMENT IN WRITING INSTRUCTION, 1985-86 

School-Community Education Program 
Program Administrator: Jack Isaacs 
Project Coordinator: Lucy Calkins 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

The Staff Development in Writing Instruction Project 
provides training in the teaching of writing to elementary school 
teachers in 18 Community School Districts (C.S.D.s 5, 6, 8, 9, 
10, 11, 13, 15, 16, 17, 18, 20, 25, 26, 27, 28, 29, and 30)- 
District superintendents, school principals, and district 
curriculum staff members selected various schools in each 
district to participate in the program. In a few of these 
schools, some teachers had already been trained in the program; 
in others, teachers were selected among a group of volunteers who 
showed interest in improving their instructional skills in 
writing process. 

In 1985-86, more than 300 teachers and some 9,000 students 
participated in the project. Teachers and their supervisors 
attended conferences and demonstration lessons conducted by 
teacher trainers and visiting consultants in topics such as 
writing as a process, improving writing through personal narrati- 
ve, revision skills and techniques, and holistic evaluation 
methods. The project also used previously trained teachers as 
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models for new participants who visited their classrooms for 
demonstration lessons. 

The objective of the project was for students of the 
participating teachers to improve their writing skills as 
measured by holistically scored writing samples. The project was 
funded for $282 thousand by the New York State Legislature to 
cover salaries for teacher trainers, consultants, and teacher 
substitutes. 

EVALUATION METHODOLOGY 

In 1985-86, project evaluation focused on analyses of 
students' writing scunples which best reflect the instructional 
skills in writing process of teacher participants. To assess 
these skills as well as students' writing ability, pupils were 
given a writing assignment on the first and last day of project 
activities. Program staff selected a representative sample of 
students' writing assignments (40 for each district) which was 
stratified by grade level. The writing samples were then 
holistically scored by a team of five raters, using scales 
developed by the project coordinator. Each of the five raters 
reviewed and scored every writing sample. For kindergarteners 
and first graders, the raters used a scale drawn from the lists 
of stages children go through in learning to write by Marie Clay 
(see Appendix A). For pupils in grades two through eight, the 
raters used a scale adapted from the Personal Narrative Writing 
Scale in Cooper and Odell, Evaluating Writing (see Appendix B). 

2 
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In both cases, students' writing samples were scored on a scale 
from one to five point? . 

Pre- and post-program holistic scores were submitted for 
evaluation. These scores were compared and correlated t-tests 
were computed to establish if achievement differences were 
statistically significant. Effect size (E.S.)* which indicates 
the educational meaningf ulness of the mean gain or loss for each 
comparison was also calculated. 

FINDINGS 

Pre- and post-program holistic scores were reported for 720 
students. A majority of the pupils' writing samples (89 percent) 
received a post -program rating of three or more score points on 
the five-point scale. Table 1 presents holistic scores by grade. 
Results showed scores to be similar across grades except for 
kindergarteners who made the lowest pre-progrcun mean score (1.8 
score points), and first-grade participants who made the highest 
post-program mean score (4.2 score points). Students in all 
grade levels achieved statistically significant and educationally 
meaningful mean gains, with the highest gains occurring in 



*The effect size, developed by Jacob Cohen, is the ratio of the 
mean gain to the standard deviation of the gain. This ratio 
provides an index of improvement in standard deviation units 
irrespective of the size of the sample. According to Cohen, 0.2 
is a small E.S., 0.5 is a moderate E.S., and 0.8 is considered to 
be a large E.S. Only effect sizes of 0.8 and above are con- 
sidered to be educationally meaningful, reflecting the importance 
of the gains to the students' educational development. 
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TABLE 1 

Students' Mean Pre- and Post-Prograin Holistic Scores^ on 

Writing Samples, by Grade 
Staff Development in Writing Instruction, 1985-86 









Pre- 


Program 


Post- 


-Program 


Difference^ 




Grade 


N 


Mean 


S.D. 


Mean 


S.D. 


Mean 


S.D. 


E.S. 


K 


60 


1.8 


.9 


3.0 


1.0 


1.2 


.5 


2.2 


1 


100 


3.0 


.6 


4.2 


.6 


1.2 


.6 


1.9 



2 


120 


2.7 


.6 


3.1 


.5 


<4 


.4 


1.0 


3 


112 


2.8 


.7 


3.3 


.7 


.5 


.5 


.9 


4 


124 


2.8 


.6 


3.3 


.6 


.5 


.4 


1.1 


5 


92 


2.8 


.7 


3.3 


.7 


.5 


.5 


1.0 


6 


88 


2.9 


.6 


3.4 


.5 


.5 


.4 


1.1 


7 


16 


2.8 


.5 


3.3 


.5 


.5 


.4 


1.3 


8 


8 


3.0 


.6 


3.4 


.6 


.4 


.4 


.8 



*Based on scales from one (low) to five (high). 

^All mean gains were statistically significant at p<.5. 

• Students in all grade levels achieved mean gains ranging 
from .4 score points to 1.2 score points. These gains 
were statistically significant and educationally meaning- 
ful. 

• Greatest gains were made by pupils in kindergarten and 
grade one. 
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kindergarten and grade one (1.2 score points). Mean gains for 
grades two through eight ranged from .4 to .5 score j^oints. 

CONCLUSIONS AND RECOMMENDATIONS 

The evaluation findings indicate that the Staff Development 
in Writing Instruction project was successful in meeting its 
objective. Students in all grade levels showed an improvement in 
their writing ability. Mean gains were statistically significant 
and effect sizes were large, indicating the educational meaning- 
fulness of the gains. The project had a greater impact on the 
performance of kindergarten and first-grade pupils than on 
students in other grade levels. Indeed, pupils in kindergarten 
and grade one achieved the largest mean gains which, as exempli- 
fied by the performance of first graders, cannot be solely 
attributed to low pre-progrsim writing ability. 

A comparison of the 1985-86 results with last year's 
evaluation findings shows that, in general, students then made 
lower scores and higher gains than this year. This fact raises 
two possibilities. First, this year's students had already been 
taught by teachers previously trained in the project, thus, they 
began the 1985-86 school year with improved writing .bility. But 
the findings also raise questions about the reliability of 
holistic inter-rater scoring since project staff did not submit 
the five sets of ratings for each pupil. In the future, the five 
ratings for each student should be furnished for evaluation in 
order to establish inter-rater reliability. Further, project 

5 
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staff should consider establishing a quantitative measure of 
project success. The following sentence, for instance, could Le 
added to the project objective: "Students will achieve a mean 
gain of at least one point on a five-point rating scale." 
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Directions for Administering Writing Sample 

• . 63404 

Please give each child a writing sample form and a pencil (only). 
Help them fill out the heading. (if necessary, fill out the 
heading .f9r them beforehand). Ask c.ildren to draw a picture 
(with their pencil only) of something they like to do. 

After five minutes, ask them to turn the paper over and write or 
pretend to write about their picture. Don't lead the children 
"into writing except to tell them, "Just put down what ever you 
can" or "just try it." Repeat the directions if necessary. 

. *Note; It seens i^jprqpriate at this time of year that first graders be allowed 
to use lined paper, (attach paper to form) . 
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DIRECTIONS FOR ADMINISTERING WRITING SAMPLE 



T«ll Students, "You are invited to t: ^ part in a soecial 

Jai i2L!n'ri'^- ""i""! " sto?yP2bout%S„,|?S?irthat 

has happened to you." Then provide each student with 2 sheets of 
paper. Suggest they take time to list possible topics. Once 
lyll ™® one, they can begin to write. Let them know 

"f? much paper as they need, and that they have plenty 
of time. Also say, "You can try writing it in rough draft or vou 
can Dust tart writing the final piece." ^ 

..mii^^K?"®.?^^*^'^®" finish early, ask them to read a book quietly 
wS«M*'!.-??"^??"'- 20 minutes interrupt the children ^ 

^^4^ writing. Say to all children, "If you had more 

ihl^ «f t "^^^ <3o next?" On another 

sheet of paper ask them to write what they would do next. Give 
them four (4) minutes to do this. 
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Evaluation Instrument 

The Writing Process Project 

Teachers College /Umbrella Programs 



Because the ultimate goal of this project is to improve the 
quality of student writing, we will holistically evaluate 
samples of writing from the first and last day of our work in a 
district in order to determine whether there has been 
significant improvement in the writing. On each of these days 
the assignment will be the same. Students will be asked to 
select a topic of personal significance to the, and to draft 
and revise their piece without input from the teacher. The 
pieces will be dated, saved and then evaluated holistically. 
These data will be gathered from at least ten classrooms in 
each of the 14 districts, and these classrooms will be 
representative of the grade levels involved in the program from 
that district. We will randomly select ten percent of the 
children from each of these 140+ classrooms, and the pre- and 
post- scores will be tabulated for those children. The goal, 
then is to have a representative sample illustrating the range 
of districts and grade levels involved. 

Methods for holistic evaluation will differ somewhat 
depending on the age group of the younster. To the best of our 
knowledge, no one has attempted to evaluate the early writing 
of primary children in this manner and therefore we have 
devised our own methods which are explained later. When the 
students are in grades 2-8, however, we can draw on and adapt 
methods described in Cooper's test, EVALUATING WRITING. 

A group of five raters will each rank all of the written 
pieces. The raters will achieve reliability because 1) they 
come from similar backgrounds and 2) they will be carefully 
trained to reach nearly perfect agreement on samples used for 
training purposes. The raters will each be a published writer, 
and they will each have a background in teaching writing. As 
Cocper suggests, the raters will not use their image of ideal 
professional writing as an absolute standard of quality, but 
will instead rate papers relatively according to the range of 
papers produced. 

Because the pieces of writing will be personal narratives, 
the raters will follow an adaptation of the guidelines from the 
Personal Narrative Writing Scale on Page 21-24 in Cooper's text 
(see attached items). 

Many of our primary school students will not be able to 
write at all«at the start of our training efforts, and so their 
growth will need to be evaluated according to developmental 
sequences of early writing. Again, a team of five raters will 



28 



63404 



hollstlcally evaluate the samples and again the raters will 
practice these evaluations with samples of writing so as to 
achieve reliability. The guiding scale, however, will not be 
the Personal Narrative Scale but instead a list of stages drawn 
from Marie Clay*s and Susan Sower *s descriptions of the stages 
children go through in learning to write (see attached items). 
We will identify the stage evidenced in the pre- and in the 
post*samples. The raters for these pieces will be persons 
trained in early childhood development and especially in 
teaching writing in the early grades. 
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CHILDREN LEARNING TO WRITE (Based on attached list from world 
renowned researcher Marie Clay) 

A Developmental Ladder 

Level 1^ Children do not appear to be able to differentiate 
between a drawing and written language. 

Level 2 Children begin to use symbols that are used in the 
culture'^s system of writing. They write strings of letters or 
scattered letters but there appears to be little sound-symbol 
connection. 

Level 2 Children label their drawings or write words on the 
page, generally using initial and final consonants only to repre- 
sent a word. 

Level 4^ Words are combined into sentences, and spellings fill 
out to include some middle consonants and vowels, also some sight 
vocabulary. 

Level 5 Children use the written code for a wide range of 
purposes: letters, poems, recipe books, signs, etc. They write 
fluently. 
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A Developtaental Ladder 



Level 1 Children are not able to differentiate between a drawing and 
written language* 

Level 2 Children are able to differentiate between a drawing and 
written language. 

Level 3 Children begin to use sj-abola that are used in the culture's 

systerD of writing. The children ^ust write stringc of letters, 
but when one asks thea to read what they wrote , the children go 
on and on. 

Level U Children try to create an alternative correspondence between 
spoken language and written language, eg* the written re- 
SPoAse may be the length of the spoken utterance according 
to the child's own reasoning. 

Level 5 Children begin using the syllabic hypothesis, i«e. using one 
symbol for one syllable. 

Level 6 Children use both the syllabic and alphabetic hypothesis (a 
relationship between le^ter8 and sounds). 

Level 7 Children "^break code/* They are now on their way to develop- 
ing their written language according to how adults use it In 
the culture* This Is when they begin to grapple with their 
Invented spellings and begin to discover the codventioaal 
spellings used in our orthography. 



A hierarchy of spelling skills: 

!• random string of letters 

2. baginning sounds only 

3* beginning and ending sounds 

U. beginning, middle, and ending sounds 

A typical pattern of learning the letters and using then: 

]• single consonants ) 

2. long vowels 

3* everything elsa in no special order: other vowel sounds, digraphs, 
consonant blends or clusters • ^ 

^ (sowers , Six Questions Teachers 

l^pj^ Ask About invented Speliinf 
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Selected elements of Cooper's scale were used for scoring. Raters scored 
the selected traits with a 2 (low), 6 (middle), or 10 (high) for style 
or voice, central figure, sequence, and theme. Raters used a 1 (low), 3 
(middle), 5 (high) for wording and syntax, spelling and punctuation. The 
best possible paper could receive these scores: 



Trait 


Score 


Style or voice 


10 


Central figure 


10 


Sequence 


10 


Theme 


10 


Wording and syntax 


5 


Punctuation and spelling 


5 




50 



The lowest scoring paper would receive these scores: 

Trait Score 

Style or voice 2 

Central figure 2 

Sequence 2 

Theme 2 

Wording and syntax 1 

Punctuation and spelling 1 

These scores were converted into the same l-^^S scale used for the K^l 

samples to facilitate comparison: 

10-18 = 1 
18-26 = 2 
26-34 = 3 
34-42 «^ 4 

42-55 = 5 
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(in Cooper and Odell, Evaluating Writing , p. 21 -24) 



Note: For the purposes of this evaluation, we will score only 
the following aspects of the student writing: 

I. B, Style or Voice 
C. Central Figure 

E. Sequence 

F. Theme 

II. A and B. Wording and Syntax 

D and E. Punctuation and Spelling 



Appendix A: Personal Narrative Wriliug Scales 

I. Genera) Qualities: 
A. Author's Role 

The author s role is the relationship of iJie author to the suh|ecl. iiiei- 
deni or person In outobiographxj the author writes abcMit liiiusHf/ 
herself. HeAhc i^ the main participant. Most of thr time he/she v ill use 
the pronouns. ) . me. we. us. In biocraphy xhv author writes aluiiit xmie 
oHirr person Ih /she ix not in\-olveii in what happens; he/she is jwst an 
ohsencr. He/she uses the pronouns, he. she. hiiu. her. it. ihey. them 

High The autlior ke;-ps !iis/hcr correct role of cither participant or 
obserwr throu^jliotit 

Miildlv In autobiojrrnphv. a feu noticeable distr.utiuK times ihe au- 
thor talks tori nitu'h about another person's actions; or. in bi- 
ofiniphy. he/she t;ilks too much about his/her own actions 

f-ow Tlic ;n,tlior talks cibmit hiniself/htTself or olbers as particijKuit 
or obscrs i r am tunc lir/shc pleases so ih.it \ nu cm lur< K i,H 
xyhetlit r it is supposed to be autobioKrapli> or JMo^raphy 
There is c oiifusion us to .luthor's role. I le/shc is not coiisisteiil- 
1) either ol>ser\er or participant 
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Charles R. Cooper 

B Style or \'oicc 

Hifih The author staler what hc/$h( ri:illy thinks an.l UcK En- 
pressing; pmciiial c*x|K*ric<iiccN. llic utiUt cciinc*: throii^h as an 
iiKlivulual. aiicl Iii.s/hcT wtirk M-riiis liki' his/l i rs niul his/Uvrs 
aldiic*. The vciia- \vc hear ni llir |iM tv really isilereyts iis. 

MidiUr The aiithdr iisi«s umcTali/alidir. or aliAlrael hiiiKiia>£e. >ehlciiii 
iiiehidiiiK persdiial details and ecaiiiiients. U'lnle tlie piece 
ni;iy lie correct, it lacks tlic pcrMinnl Icmch The v<iiee seems 
hhiiid, careful, a little flai, and iiut ver> iiileri*sliiij' 

Lo\L We don't really lit-.ir a reciiKnizahle \iiice in the piece. *I1ie 
style seems flat and lifel(*ss. 

C. Central Figure 

Details about the central figure make hun/hcr seem "real." The char- 
acter is described physic:ill> and as a person. 

High The central figure is described in sueli detail thai he/she is 
always "real" for you 

Middle The central character can be "seen," but is not as real as 
he/she could be. 

Low The central character is not a real Wvxng person; he/she ii just 
a name on a page. You cannot sec him/her or understand 
him/her. 

D. Background 

The setlin;; of the action is detailed so that it seems to Rive the events 
a "real" place in which to hnppen. 

//f/r/i TJie actioii occurs i)i a wdl-detniled plnce th:it you cnn alincs! 
see. 

Middle Sometimes the setting seenn vivid and real, hut sdinctiinev 
the aetitin is juM liappi*ning, and you arc not rc.dlv awari- cii 
vh.it the setting is. 

l^w The action occurs without any detaih'd setting Vou sec* the 
action, hut \ou cannot see it ui a eerlnin pl.iu* 

I"- .Sc'jiicnci' 

'i he orch-r nJ r\rui^ is eli ar. p\in;; t!.f ir^n'u r a pu c im- \ h-w tlu 
M*<pu»nee ni iiieidt'iits 

//ixr/i Tin- cirder cif events is always elcMr in you c \ if al liiiii-s the 
author iiiieht talk aiicnit the past or tlir (ittuic 

Middlf A few tnne.s it is not elear uliieh e\ eiil h:i)inc ;ie(f fir-.l 

hm \\m c.inno) fi>;ure omI w Iiiih ciil ionics firs! oi f»n-. 

after an\ other c'\*<*nt 

I' "f'heiiM- 

The' author elinnses the iiieidc iits and ch l.iiK for some le.ison. 'I hen 
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Mid^llc You can set- why tl.c anfhors «.l,icct is u„p„r,anl to l.in./hcr 

but ,1 .s not as clearly slated or implied as it could be 
Low You cannot figure out why the subject is important to the au- 

II Diction. Syntax, and Mcclianics 
A. Wording 

Hiph Words are employed in . unique and interesting «•a^•. While 

S.?L ^* '""PPropriate. the author 

»eems thoughtful and imaginative. 

Middle Common, ordinary- ^vords are used in the same old way. The 
the^h^r '^^'T^'^ expressions. Tl.e author, 
SMh! A^^' ""„y " »' different that 
»uf*rSt1nf'"^ 

/-ow The word choice is li,„itcd and immature. Sometime. ».-„rd. 
are etcn used i-icorrcctiy-tlic wrong word is „s„| 

B Syntax 

//.A-'. The ^.•ntc•nce* ar. x ari.xJ in It ng,!, and structure The m.tl.or 
sliows a confident control of sentence structure The paper 
reads smoothly from sentence t.. scnti^co. Thi^c arc no rim- 
tOKCIlKT seiitenm or senlfin,. frapi.icnts. 

>/irW.V Ti.r :H.tl.,.rs|m„»s,.:„c,.,„„r f ., nt,-.,ir Mru, tuM-aml ouK 

..(•.•;.M.,„:.|,j wr,t,. s,.,.:rn, r vi.u „ :.uk« :,r,l ..r p,.//ln„. 
Aliiiiisi no nui-iiiis and lr;iKiuc ii:s 

h'u Many problems with srnU nir structure. Sonlintv-s j.re sU„rf 
and simple u, sln.ct..re. .so„-u hat ch.ldl.U- and rq,eiit.„us 
tlicir patterns. Ihrro may be nin ons and fraf!n)rais 

Is.inr 

H>d. Th,T.. :,r,. n.. o|,vi„„s ...rors „, .k.-.k.. TI... :.„i1. : sl.„us 
l.< AI..- fm,ul>.u « it • si..,„l..r,K „f ,^I„,.,| „.,,„,.„ ,. 

Mi,l,llr A I,.,, ,^r.^s i„ „s;,k,. :,p,„,,. i„ ,,; .>l...« n.,: ll,,- .„„iM„ 

lias not ,,„,te |>« i. ciiusistint m usin^ ^t;^ml:lrtl forms 
l^m Hic »vritii,K is full „f ..s..,;.,. ,.r„,rs 
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Charter R. Conpc 

D Puncluatioti 

Hif^li yUv author coiisis(rntK uses :)})|)ii)|)ri.itr pitnctuatioi) 
Miildic NJnst of llic tiiiir llit- wrilrr piiiictuatrs forri-cll> 
IjOw The writing cotit:iitis tiuiny ]Yiinctu:itioi) errors 

E. S))cllmg 

l^igh All wordi arc Npt'lli*(l c orrittly 
Middle A fi-xv words :irr missprllal 
Imw Many words arc tnissprnixl. 



I. General Qualities: 

A. Author's Hole 

B. Style or Voice 

C. Central Figure 

D. Background 
£. Sequence 
F. Theme 



Analytic Scale 



Keader. 
Low 



2 4 

2 4 

-2 4 

2 4 

2 4 

'2 4 

II. Diction, Syntax, and Mechanics: 

A. Wordtng I 2 

D. Syntax 1 2 

C. U«age I 2 

D. Punctuation 1 2 
£. Spelling 1 2 



Pa|>cr_ 



Middle 

6 
C 
6 
6 
C 
6 

3 
3 
3 
3 
3 



8 
S 
3 
8 
8 
8 

4 
4 
4 
4 
4 



nii;h 

10 
10 
10 
10 
10 
10 



Total 





Dichotomous Scale 




Reader 




yns 


NO 




> 




Author's role cotibt»tetit 
Interesting pcrMnal voice 
Theme clearly presented 
Background rich and Mipportivc 
Sequence of evenU clear 
Central figure fully developed 


1! 




Wording uni(|ue and*dcvclo|»cd 
Syntax correct and varied 
Usage erron few 
Punctuation errors few 
Spclhng errors few 
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STAR LABS, 1985-86 

School-Community Education Program 
Program Administrator: Jack Isaacs 
Project Coordinator: Samuel Storch 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

STAR LABS is a staff development project designed to assist 
elementary and high school teachers improve their skills for 
science instruction. The project was implemented by Community 
School Districts (C.S.D.s) 21 and 22 in collaboration with the 
Edward R. Murrow High School. 

In 1985-86, 40 teachers from C.S.D.s 21 and 22 participated 
in the program. They were selected by their school principals 
based on their need to improve their science instruction. 
Project activities took place at the E.P. Hubble Planetarium in 
Edward R. Murrow High School and consisted of two components. 
First, teacher participants took part in three five-hour training 
sessions at the planetarium to learn concepts of astronomy and be 
provided with appropriate lesson plans and materials to use in 
their classrooms. Secondly, participating teachers and their 
students visited the planetarium to view shows specifically 
designed for each grade level, emphasizing the earth's relation- 
ship to the sun, moon, planets and other star systems. Finally, 
a third project component was to take place at C.S.D. 28, using a 
portable planetarium to train teachers and involve students* 
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This component, however, was not undertaken. Funds for $37 
thousand were provided by the New York State Legislature to 
acquire audiovisual material, general instructional supplies and 
support after-school teacher workshops. 

The project objective was for teacher participants to demon- 
strate knowledge in science and techniques, including planning 
appropriate astronomy lessons, selecting relevant material and 
involving students in hands-on activities. Teacher performance 
in these areas was measured by a program-developed test. 

EVALUATION METHODOLOGY 

Evaluation of the project focused on the analysis of 
participants' scores on a program-developed test (see Appendix 
A) . The test consists of 20 multiple-choice items on concepts of 
astronomy and selection of appropriate teaching materials and 
activities for different grade levels. The test was administered 
at the beginning and end of the training period. In addition, 
O.E.A. personnel visited the Planetarium. 

FINDINGS 

Complete test scores for 30 participants were submitted for 
evaluation (see Table 1). Overall, teachers achieved a 30 
percent ii crease at posttest. Mean pretest score was 7.6 points 
(38 percent correct) and mean posttest score was 13.6 points (68 
percent correct). A comparison of test scores by district shows 

2 
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TABLE 1 

Participants' Mean Raw Scores^ on a Project-Developed Test, 

by District 
STAR LABS, 1985-86 



District 



Pretest 



Raw Percent 
N Score Correct 



Posttest 



Raw Percent 
Score Correct 



Gain 



Raw Percent 



C.S.D. 21 


13 


7.1 


35.4% 


12.5 


62.7% 


5.4 


27.3 


C.S.D.22 


17 


8.1 


40.6 


14.6 


73.2 


6.5 


32.6 


TOTAL 


30 


7.6 


38.0 


13.6 


68.0 


6.0 


30.0 



^Perfect Raw Score = 20. 

• Overall, participants achieved a gain of 30 percent. 
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that teachers at C.S.D* 22 performed better than teachers at 
C.S.D. 21. 

In addition to these quantitative findings, the Director of 
the Edwin Hubble Planetarium indicated that students were well 
prepared for their planetarium visit and "loved it." Several 
letters^ written by teachers^ corroborate his statement. 

CONCLUSIONS AND RECOMMENDATIONS 

The evaluation findings show that STAR LABS was a successful 
project having an impact on the performance of participating 
teachers. Additional qualitative findings indicate that their 
students also benefitted from the program. In the future, 
however, project staff should consider including in the objective 
specific quantitative criteria for successful program completion. 
The following sentence could be added: "For 80 percent of 
participants, posttest scores will show an improvement of at 
least 25 percent over pretest scores." 
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E=P„ HUBBLE 
PLmETRHJIJin 

TEkCHSR CTAniA TICN INSTiaiMENT 



E. R. MURRCW HIGH SCHOOL 

ISOO AVENU6 U, aPOCKLYN, NEW YCRKIISSO 

718 flvsssa-saaa exT.aixy 



Inatructiapai flease chooae the answer which best cooipletes the statement 
or ar 'era the qjiestion. Mark this answer on the separate answer sheet. 

!• What is represented by the center of the cofflmon. circular star map? 
(a; the North Celestial Pole (b) the horiaon (c) the tenith (d) the 
meridian. ' 

2« Which of the following materials would be of greatest value In stidy^ 
Ing the Dhases of the moon in preparation for a planetarium demonstra- 

(a; "oon globe, earth globe, chart of phases (b) ball-ani-stick 
models, empty sUda projector (c) calendar with lunar phases, diagram 
of phases, protractors and rulers (d) orrery and flashlight. 

3« Which of the following factors is most responsible for a student »s 
^jyj*^ to see many stars from outside his or her home? (a) air 
pollution (b) horiacns blocked by buildings and trees (c) poor 
dark adaptation (d) waste lighting. 

^' /5^^v*^„*!lf '<»p-owing would be visible first after dark? (a) Venus 
(b; the Nortn Star (c) the Orion NebuU in winter (d) the Airiromeda 
Gala:;^* 

5. J'Jich of the following materials would most graphically demonstrate 
the distances bf tween planets in the solar system? (a) pencU, roll 
of adding raach^ie tape, rulers, table of distances (b) chart of the 
solar system, pictures of planets, table of distances (c) model of 
the solar system, table of distances, globes of Earth and moon (d) 
ai visit to the planetarium* 

6« Which is the correct statement? fa) The sun rises in the east, (b) ""he 
sun is ai the meridian at noon, (c) The sun sets in the west, (d) T .e 
time the sun is above the horiaon does not vary. 

7. Which statement is incorrect? (a) The moon can be found during daylight 
on nearly every day of the month, (b) The orbit of the Earrth bringsit 
cloder to the sun during our winter, (c) The North Star is the brightest 
star in the sky. (d) A brighter star may act-ially be more distant than 
a fainter star. 

^ * "changing time" for young children, which of 

the following would be a good preparation for a class before a visit 
to the planetarium? (a) Let children check the time of surrise and 
sunset at home using the newspaper times and a watch (b^ give the 
children a diagram showing the meridian and horiion, and the sun's 
rSi^^SI*^ (c) teach the children to tell time using a large clock 
iSS .^tL^i^' diagrams of cardinal points, sunrise aSdSJSSIt piiSs, 
and a aunrise-sunset table. *^ ' 
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% Which of the foUnwlng is the most appropriate for x fifth grader 
to use fhen leanung the constellations? (a) a small telescope 
{t) a celestiAl globe (c) an umbrella with the stars paiited on 
the inside (d) a pair of binoculars aid a star map* 

10. Wiich of the foUowlng is the best clasarooo demonstration to show 
the changing position of the Big Dipper during the night? (a) a 

Northern Skies with Polaris at the center (b) rotating 

stars painted on the Inside (d) a round flask with stars on the out- 
side and blue tinted water half-filled on the inside. 

11. Which concept is most appropriate for a kindergarten class? (») Are 

°S ^ *° the Zod2c (c) Locating thf 

Tfopic Parallels (d) How we tell tiae by ;he Stars. 

Sf^J ?J following is ngt usually a concept or topic best taught 
Wh^^'^'^TJ*^ ^« °^ thrstars ^) the midS^ht 

wn (c) the biases of the moon (d) the characteristics of the planets. 

Si/y.lJfio^f^f^l!?? arrangements where students 

will face south primarily? (a) seating convenience is enhanced 
^b; most seasonal sky changes occur in the south (c) the entire 

class can be taught at once with eye contact to the Instructor 

Cdj we Uve In the northern part of the country. 

U. Wjich of the following magaslnes is most suited for elementary and 
Junior high school^ntermediate school students? (a) Skv and Tel^aeoni. 
(b) Astroiowy (c) Odvssev (d) The Griffith at^^rrrlJ .- ^^ 

I^L^A mechanical model of sun-Earth-«oon often 

f^n S^ t P^^^^rium (b) an orrexy (c) an eidur- 

aneon (d) a celestial globe. 

16. Which of the following best describes the plaietaHum? (a) a mo^e 
(o) a mechanical model of the solar system (c) a video or laser 

tSI1S,'erorplj;?oI'esr °' * ^^^^ '^'^ 

S^SsJ^fff ff^h causes .odiac signs to change over tre ages 
of history? (a) rotation (b) revolution (3) precession (d) nuta>,ion. 

iStn il *"f^* between the moon and sun as seen from the Eaz-th 
When the moon is at first quarter? (a) 45" (b) 90« (c) 180« (d) 23^«. 

??tt^?v.w i«?>ortant function of a telescope? (a) gathering 
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ANSWER KEY 





C 


2. 


B 


3. 


0 


4. 


A 


5, 


A 


6. 


B 


7. 


C 


8. 


A 


9, 


D 


10, 


C 


11. 


A 


12. 


0 


13. 


B 


14 . 


C 


15. 


B 


16. 


0 


17. 


c 


18. 


B 


19. 


A 


20. 


A 
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MATHEMATICS IMPROVEMENT PROGRAM, 1985-86 

School Community Education Program 
Program Administrator: Jack Isaacs 
Project Coordinator: Aurora Larocca 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

The purpose of the Mathematics Improvement Program is to 
provide support services to teachers in elementary and interme- 
diate schools in Community School Districts (C.S.D.s) 14 and 32. 
These services include pupils diagnostic information and 
prescriptive techniques designed to improve student growth in 
mathematics. In 1985-86, the program served some 116 teachers 
ana 3,420 pupils in grades two through seven in 12 elementary and 
five intx^rmediate schools in C.S.D. 32. 

The participating schools were selected by the District 
Mathematics Coordinator and by the school principals. In 
addition, C.S.D. 14 teachers were selected by school principals 
to participate in a lessen plan writing activity. Doth district 
and Board of Education staff trained participating teachers. 
Staff development workshops focused on the development of 
effective prescriptive techniques. In addition^ computers were 
used to provide individual information about pupils needs, 
progress, and mastery of specific mathematics skills. The 
project objective was for 55 percent of participating pupils to 



ERIC 



45 



63411 

achieve on or above grade level in mathematics as measured by 
1985-86 citywide achievement tests. 

The New York State Legislature contributed $46 thousand to 
support project activities. The bulk of these funds were used in 
C.S.D. 32 for instructional supplies, data processing eguipnent, 
and equipment repairs. The remaining funds were used by teachers 
in C.S.D. 14 to develop mathematics curricula materials. The 
project also shared the resources of the Comprehensive Instruc- 
tional Management System (CIMS). 

EVALUATION METHODOLOGY 

Evaluation activities focused on the analysis of student 
outcomes on the Stanford Diagnostic Mathematics Test. Student 
scores for April 1985 were compared with those obtained in April 
1986. All raw scores were converted to normal curve equivalent 
(N.C.E. )* scores which express student performance relative to a 
national norm. U.S. Department of Education Evaluation Model A 
was used to determine project impact on student achievement in 
which mean N.C.E. gains are attributed to project services. 
Correlated t-tests were computed to establish if achievement 



*N.C.E. scores are similar to percentile ranks, but unlike 
percentile ranks, are based on an equal-interval scale. Normal 
curve equivalent scores are based on a scale ranging from 1 to 99 
with a mean of 50 and a standard deviation of approximately 21. 
Because N.C.E. scor<^s are equally spaced apart, arithmetic and 
statistical calculations such as averaging are meaningful; in 
addition, comparisons of N.C.E. scores may be made across 
different achievement tests. 
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differences were statistically significant. Effect size (E.S.)* 
which indicates educational meaningfulness of the mean gain or 
loss for each comparison was also calculated. 

FINDINGS 

Complete test scores were sulMtiitted for 940 students, 
representing a sample of about 27 percent of the total number of 
participating students. Table 1 presents findings on student 
performance on the Stanford Diagnostic Mathematics Test by grade. 
All gains were statistically significant and, in general, educa- 
tionally meaningful. Overall, pretest mean score was 23.8 
N.C.E.s, posttest mean score was 35.7 N.C.E.s, for a mean gain of 
11.9 N.C.E.s. Gains ranged from 4.7 to 21.2 N.C.E.s with fourth 
graders achieving the largest gain. Second- and seventh-grade 
pupils made the largest pretest scores but achieved the lowest 
gains. Effect size was large for all grades except grades two 
and seven. 

Table 2 presents the percentage of students who met the 
project objective (55 percent of participating pupils will 
achieve on or above grade level in mathematics). Only about 17 
percent of students scored on or above grade level. Grades two 

*The effect size, developed by Jacob Cohen, is the ratio of the 
mean gain to the standard deviation of the gain. This ratio 
provides an index of improvement in standard deviation units 
irrespective of the size of the sample. According to Cohen, 0.2 
is a small E.S., 0.5 is a moderate E.S., and 0.8 is considered to 
be a large E.S. Only effect sizes of 0.8 and above are consi- 
dered to be educationally meaningful, reflecting the importance 
of the gains to the students' educational development. 
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TABLE 1 

Students' Mean N.C.E. Scores on the 
Stanford Diagnostic Mathematics Test, by Grade 
Mathematics Improvement Program, 1985*86 



Pretest Posttest Difference^ Effect 

Grade N Mean S.D. Mean S.D. Mean S.D. Size 



2 


246 


30.0 


13.2 


37.3 


18.3 


7.3 


18.2 


.4 


3 


173 


21.6 


11.9 


34.4 


13.4 


12.8 


15.2 


.8 


4 


117 


15.8 


14.4 


37.1 


13.1 


21.2 


12.7 


1.7 


5 


178 


22 . 1 


11.5 


35.5 


14.0 


13.4 


12.6 


1.1 


6 


133 


23.0 


11.2 


37.0 


14.3 


14.0 


13.6 


1.0 


7 


93 


26.0 


13.5 


30.8 


10.0 


4.7 


11.4 


.4 


TOTAL 


940 


23.8 


13.3 


35.7 


14.9 


11.9 


15.5 


.8 



^These gains were significant at p<.05. 



• Students at all grade levels achieved statistically 
significant gains, ranging from 4.7 to 21.2 N.C.E.s. 
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Students Meeting Project-Set Criterion^, by Grade 
Mathematics Improvement Program, 1985*86 



Grade 


N 


Meeting 

N 


Criterion 

% 


2 


246 


60 


24.4% 


3 


173 


19 


11.0 


4 


117 


18 


15.4 


5 


178 


32 


18.0 


6 


133 


30 


22.6 


7 


93 


4 


4.3 


TOTAL 


940 


163 


17.3 



*Fifty-five percent of participating pupils will achieve on or 
above grade level in mathematics. 



About 17 percent of participating students met the 
project-set criterion of success. 
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and six had the largest proportion of students meeting the 
project-set criterion of success while grade seven had the lowest 
number of successful students. 

CONCLUSIONS AND RECOMMENDATIONS 

The evaluation findings indicate that although student 
participants at all grade levels achieved statistically signifi- 
cant gains ^ the program objective was not met. As formulated, 
this objective anticipated that 55 percent of pupils would 
achieve on or above grade level in mathematics but only about 17 
percent of participants achieved this objective. As indicated in 
previous evaluation reports, the objective has been too ambitious 
and the principal reason for the lack of project success in the 
last years. If student performance is measured, instead, by 
their improvement in mathematics by differences between pretest 
and post test scores rather than by their post test score alone, 
the project would have a more realistic objective. In 1985-86, 
for instance, mean gains were not only .^arge but also educa- 
tionally meaningful in all grades except grades two and seven. 
In the future, project staff should modify the project objective 
to include a mean gain, for instance, "students will achieve a 
mean gain of at least five N.C.E. scores." 
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REASONING/THINKING SKILLS PROGRAM, 1985-86 

School-Community Education Program 
Program Administrator: Jack Isaacs 
Project Coordinator: Ira Ewen 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

The Reasoning/Thinking Skills Program was designed by the 
Division of Curriculum and Instruction in collaboration with the 
Queens College Center for the Improvement of Education, selected 
Community School District (C.S.D.) superintendents and the 
Citywide Umbrella Bureau. The project provides instruction on 
thinking and reasoning skills models to administrators, teachers, 
and curriculum specialists. Participants also receive assistance 
for developing and directing reasoning skills projects in their 
districts. The goal of the project is for participants to be 
able to ch^^«^** the most effective pedagogical approaches to 
address the particular educational needs of their district. 

The project was first implemented in 1985-86. All New York 
City Community School Districts were invited to select five 
representatives to participate in the program. They were 
selected among administrators, supervisors, curriculum officers 
and teachers who were willing to participate in the project and 
in the training design. A total of 12 districts were repre- 
sented. 
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Project activities were undertaken in two steps. In the 
firsts participants attended seven weekly training seminars to 
explore and analyze major thinking and reasoning skills models, 
including those of Bloom, Feuerstein, Guilford, Renzulli, Lipman, 
and Ennis. During the second part of the project, participants 
developed and sulMnitted proposals for reasoning skills programs. 
Project staff consisted of one academic consultant from Queens 
College. The New York State Legislature provided $15 thousand to 
cover project expenses. 

The objective of the project was for participants to 
demonstrate knowledge of the major models for reasoning and 
thinking skills education based upon contemporary educational 
literature and practices. 

EVALUATION METHODOLOGY 

Project impact was assessed by an analysis of participants' 
scores on a project-developed test. The test, consisting of 21 
multiple-choice items, was administered on a pre- and posttest 
basis at the beginning and end of the first phase of project 
activities (see Appendix A). 

FINDINGS 

Pretest and posttest scores were reported for 48 partici- 
pants from nine districts identified by letter rather than by 
number (see Table 1). All participants made gains from pretest 
to posttest. Overall, pretest mean raw score was 9.4 points 
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TABLE 1 

Participants' Mean Raw Scores* on a Project-Developed Test, 

by District 
Reasoning/Thinking Skills Program, 1985-86 





District^ 


N 


Pretest 


Posttest 


Gain 


% 


Raw 
Score 


Percent 
Correct 


Raw 
Score 


Percent 
Correct 


A 


c 
D 


1ft n 
iU . U 


4 / . D« 


19. 0 


90 . 5% 


ft ft 

9.0 


A ft c 


B 




10.8 


51.4 


17.6 


83.8 


6 . 8 


32. 4 


c 


6 


10.5 


50.0 


16.7 


79.5 


6.2 


29.5 


D 


4 


8.5 


40.5 


13.7 


65.2 


5.2 


24.7 


F 


4 


6.7 


31.9 


15.7 


74.8 


9.0 


42.9 


6 


7 


8.3 


39.5 


14.4 


68.6 


6.1 


29.1 


H 


7 


8.1 


38.6 


17.4 


82.8 


9.3 


44.2 


J 


5 


11.0 


52.4 


18.6 


88.6 


7.6 


36.2 


K 


5 


10.4 


49.5 


17.2 


81.9 


6.8 


32.4 


TOTAL 


48 


9.4 


44.6 


16.7 


79.5 


7.3 


34.9 



^Perfect Raw Score = 21 

^he coordinator of the project substituted lettt 3 for district 
numbers. 



• Participants achieved a mean gain of 34.9 percent. 

• Districts A, F, and H outperformed other districts. 
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(44.6 percent correct responses), posttest mean raw score was 
16.7 points (79.5 percent correct) for a mean gain of 7.3 points 
or 34.9 percent. Pretest scores by district ranged from 6.7 to 
11 raw score points, posttest scores had a wider range, from 13.7 
to 19 raw score points and gains ranged from 5.2 to 9.3 raw score 
points (24.7 to 44.2 percentage points). 

coNCLUsic::s and recommendations 

Evaluation findings indicate that the Reasoning/Thinking 
Skills Program was successful since participants achieved a mean 
gain of about 35 percent. The fact that the gain is large, shows 
that participants benefited from their participation in the 
program. In the future, however, project staff might consider 
revising the objective to Include quantitative criteria for 
successful progr£un completion. For instance, the following 
sentence could be added to the objective: "80 percent of 
participants will achieve a gain of at least 30 percent." 




54 



APPENDIX A 



63417 



REASONING/THINKING SKILLS PROJECT 
EVALDATION TEST FOR PARTICIPANTS 



Choose the response that best answers the question from a 
Reasoning/Thinking Skills perspective. 

1. Whiqh of the following Is least specific in its relation . 
to teaching Reasoning/Thinking Skills? 

a« American educational trc^dition 

b. electronic media 

c# authoritarian structures and society 

d. need for responsible citizenship 

2. Lipman and de Bono agree on 

a^ discussion as the basis for training 

b. the topic specificity of materials 

c. the need for indeptb involvement in discussions 

d. the need for creativity 

3. Which of the following is least likely to occur in a 
Reasoning/Thinking Skills lesson? 

a. reading texts 

b. learning facts 

c. brainstorming 

d. problem solving 

4. The main distinction in Bloom's Taxonomy is between? 

a. left*brain right-brain 

b. verbal and guantative 

c. cognitive and affective 
f ' • fact and opinion 

5. Which pairs does Ennis* taxonomy not group together? 

a. seek reaf-^ns/try to remain relevant to the main project * 

b. possibility of corrobation/use of established procedures 

c. expertise/agreement among sources 

d. class logic/interpretation of statements 

6. Weinstein and Cannon differ from the other taxonoraists 
calling attention to 

a. formal reasoning 

b. interpersonal reasoning 

c. philosophical reasoning 

d. Informal reasoning 
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7. Project Intelligence is associated with 

a« Perkins 

h. Whimby 

c« Guilford 

d« Paul 

S. Sternberg does not endorse 

a* Llpman 

b. de Bono 

c. Jones 

d. Feuer stein 

9. Which of these programs -does not strongly emphasize 
classroom strategies 

a. Instrumental Enrichment 

b. Philosophy for Children 

c. Mastery Learning 

d« Hanson, Silver Strong and Assoc. 

10. Which program is least culture bound? 

a. Structure of Intellect 
b« Instrumental Enrichment 

c. Mastery Learning 

d. Philosophy for Children 

11. Which intellectual ability is not isolated as basic 
in the Structure of Intellect approach? 

a« cognition 

b. divergent thinking 

c. formal reasoning 

d. evaluation 

12. Which attitude is held in common by Instrumental Enrichment 
and Mastery Learning? 

a« teachers must model reflective thought 

b. spatial reasoning is a key for teaching the culturally 
deprived 

c. the individual student must learn at his or her own 
pace 

d. reasoning skills must grow out of the standard 
curriculum 

13* Future Problem Solving is not necessarily related to 
which subject area? 

a. mathematics 

b. reading 

c. social studies 
d* science 
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14 Philosophy for Children is based on 

a. children's versions of classic philosophical texts 

b. discussion of contemporary social problems 

c. realistic portrayals of children in novels 

d. problems growing out of the pupils classroom and 
home experiences 

• 

15. Talents Unlimited most characteristically enables 
gifted children to 

a. develop discussion skills 

b. do independent research 

c. read the classics 

d. put on talent shows 

16. Which area of writing least involves reasoning skills? 

a. editing for grammar 

b. editing for style 

c. editing for consistency 

d. editing for organization 

17. Reasoning in mathematics characteristically does not involve 

a. deductive logic 

b* analysis of language 

c. inductive logic 

d. analogical reasoning 

18. Which theoretician has the least to say about reasoning 
and reading? 

a. Feuerstein 

b. Lipman 

c. Whimby 

d. Guilford 

19. Reasoning in Social Studies is least likely to occur in 

a. discussing the accuracy of historical reports 

b. analyzing the role of environment in economics 

c. applying past lessons to present problems 

d. preparing noces for rese'arch projects 

20. Computer programming requires the mastery of 

a. if-then reasoning 

b. nand-nor logic gates 

c. algorithms 

d. BASIC 

21. Scientific reasoning does not invariably require 

a. deductive logic ^ 

b. inductive logic 

c. reasoning by analogy 

d. mathematical reasoning 
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ANSWER KEY 



1. a 

2. d 

3. b 

4. c 

5. b 

6. c 

7. a 

8. b 

. 9- a 

10. b 

11. c 

12. c 

13. a 

14. c 

15. b 

16. b 

17. c 

18. z 

19. d 

20. a 

21. d 
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ARTS IN GENERAL EDUCATION (AGE), 1985-86 

School-Community Education Progrcun 
Program Administrator: Jack Isaacs 
Project Coordinator: Elton Warren 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 



PROJECT DESCRIPTION 

The Arts in Gene'^al Education (AGE) Project is designed to 
assist teachers in pla..ning and integrating arts into their 
regular classroom curriculum. The goal of the project is to 
improve the instructional skixls of teacher participants so that 
they can provide more stimulating learning experiences to 
students. 

In 1985-86, 62 elementary and high school teachers from 13 
Community Schools Districts (C.S.D.s) 2, 3, 6, 8, 10, 12, 13, 19, 
22, 24, 26, 27 -»nd 29), La Guardia High School of Music and Art, 
and John F. Kennedy High School participated in the program. 
Those teachers willing to participate in the progrcun were 
selected by school principals. Participants attended a series of 
workshops on music, dance, folk ait, and other fine arts. These 
were: 

• "Music, the Enabler in the Learning to Learn Process," 

^ "Liberties with Liberty" (Museum of American Folk Art), 

• "A 'Miss Liberty' Banner Workshop," 

• "The Many Languages of Dance: Communicating With, 
Through and About Movement" (Joyce Theater), and 

• "Movement and Music: Catalysts for Nurturing Creati- 
vity." 
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Art specialists conducted the workshops which included demonstra- 
tion lessons, ballet and modern dance classes, stage presenta- 
tions and performances. Conferences were provided for principals 
to support staff development training. Students also attended 
dance performances at the Joyce Theater. The objective for 1985- 
86 was for participating teachers to demonstrate their ability to 
integrate knowledge regarding the arts into the basic instruc- 
tional program. 

The project received $54 thousand in funding from the New 
York Stace Legislature. Project staff sought additional resour- 
ces, frori foundations and other art-oriented organizations. For 
example, the Metropolitan Opera Guild Education Department in 
cooperation with the New York City Board of Education Arts in 
General Education Network and C.S.D. 6 organized the 1985-86 Met 
Opera Teacher Workshop series. Twenty-one AGE teachers partici- 
pated in this program which is not evaluated in this report. 

EVALUATION METHODOLOGY 

Evaluation activities focused on three areas: 1) teacher 
response to the AGE workshops; 2) the impact of the workshops on 
teachers' instructional practices as assessed by their supervi- 
sors; and 3) the impact of the program on the participating 
schools as assessed by the principals. 

Teachers' response to the workshops was measured by a 
workshop Evaluation Form distributed at each workshop (see 
Appendix A). Teachers were asked to rate their agreement with 

2 
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five statements about the workshop on a five -point scale from 
"strongly agree" to "strongly disagree," and to describe the 
positive and negative features of the workshop. The highest 
score possible was 25. 

The impact of the workshops on the teachers' instructional 
practices was assessed by a Teacher Survey completed by the 
teachers' supervisors (see Appendix B). The survey consisted of 
five statements about the teacher's use of art activities in the 
classroom and asked the supervisor to rate how frequently the 
statement was true of the teacher. To facilitate aria lysis, a 
numerical value was assigned to each response: never=0, 
seldom=l, sometimes=2, and f requently=3. Percentages for each 
response are reported, as well ss a mean score for the item. 

The impact of the AGE program on the participating schools 
was assessed by a Principals' Survey (see Appendix C). Princi- 
pals were asked to respond to four questions rating the extent of 
project impact on teachers and students at their schools and to 
cite one specific excunple of AGE's contribution to the school 
program. To facilitate analysis, a numerical value was assigned 
to each response: "a" (most positive) = 3, "b" (less positive) = 
2, "c" (negative) = 1, and "d" in Item 3 (most negative) = 0. 
Ratings are reported by percentages for each response and a mean 
score for each item. 
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FINDINGS 

Workshop Evaluations 

About a third (ill) of workshop evaluation forms were 
submitted for analysis. The findings are presented in Table 1. 
Individual workshops received ratings ranging from 93.2 to 99.6 
percent. Overall rating was 96.1, a highly positive teacher 
response to workshop activities. In the comments, most teachers 
uniformly described the workshops as "stimulating, motivating, 
and creative." Several teachers stated their interest in having 
more or longer workshops. There were very few negative comments. 
Some high school teachers indicated that training activities and 
materials were mostly designed for elementary school children. 
Teacher Surveys 

Fifty Teacher Surveys were submitted for evaluation. For 
most of the teachers (26) this was the first year of program 
participation, 12 had been in the project for two years, and four 
for three or more years. Eight surveys did not specify the 
number of years participants had been in the program. Most 
teachers were reported to perform the target activities 
"frequently" (see Table 2). Ninety-eight percent of the teachers 
received the highest rating for encouraging students* creative 
efforts, and 96 percent of the classes surveyed showed that 
students responded enthusiastically to arts activities. Instruc- 
tion in basic arts concepts was the least frequent activity. No 
teacher received a rating of "never." 

4 
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TABLE 1 

Teachers' Evaluations of the Project Workshops 
Arts in General Education, 1985-86 



Workshop Title 


N 


Mean 
Score^ 


Mean 
(*) 


X • nusxc f ^ne cnaDxex xn une 
Learning to Learn Process" 


25 


23.3 


93.2% 


II. "Liberties With Liberty" 


15 


23.4 


93.6 


III. "A 'Miss Liberty' Banner" 


28 


24.2 


96.8 


IV. "The Many Languages of Dance" 


18 


24.9 


99.6 


V. "Movement and Music" 


25 


24.3 


97.2 


TOTAL 


111 


24.0 


96.1 



^Highest Score =25 



• On the whole, the workshops had a 96 percent rating. 
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TABLE 2 

Supervisors' Ratings of Teachers' Use of Arts in Instruction^ 
Arts in General Education^ 1985-86 



Frequency 



Item Seldom Sometimes Frequently Mean^ 



1. Lesson plans indicate 
integration of arts 
activities in in- 
struction 2 


(4%) 


7 


(14%) 


41 


(82%) 


2 


.8 


2. Schedules external 
arts activities for 
students 




11 


(22%) 


39 


(78%) 


2 


.8 


3. Provides instruction 
in basic arts con- 
cepts 1 


(2%) 


15 


(30%) 


34 


(68%) 


2 


.7 


4. Students demonstrate 
enthusiastic response 
to arts activities 




2 


(4%) 


48 


(96%) 


3 


.0 


5. Encourages students' 
creative efforts 




1 


(2%) 


49 


(98%) 


3 


.0 



= 50 



bNo item received a rating of "Never". 

^Seldom = 1, Sometimes = 2, Frequently = 3 

• Most teachers were reported to perform all activities 
frequently. 
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Principals ' Surveys 

Principals' Surveys were received from 32 school principals 
(see Table 3). On the whole, principals were generally positive 
about the impact of the AGE project on their schools. About 85 
percent of them reported that AGE had benefited classroom 
instruction at their schools and two-thirds thought that both 
teachers and students benefited from A6E*sponsored external 
classroom arts programs. Item 2 ("AGE teachers shared their 
project-related experiences with other teachers at their 
schools") received the lowest rating. Eighty-seven percent of 
the principals believed that teachers at their schools were 
interested in participating in future AGE workshops. The 
principals most common examples of AGE*s contribution to the 
school program were: increased rapport aunong teachers, active 
teacher involvement in the organization of school festivals, and 
increased motivation among AGE participants. 

CONCLUSIONS AND RECOMMENDATIONS 

The evaluation findings for 1985-86 show that the Arts in 
General Education was a highly successful project. Teachers 
found project activities "stimulating, motivating and creative" 
and gave the workshops an overall rating of 96.1 Supervisors 
reported that teachers encouraged student creativity and that 
most of them responded well to the teachers' efforts. Principals 
also indicated that the progrcun benefited their schools. Thus 
the program met its objective of helping teachers to integrate 
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TABLE 3 

Principals' Response to Project Activities^ 
Arts in General Education, 1985-86 





Statements 


Choices 


Perceiit 


Mean 



1. 


Skills learned in AGE 


a. 


Considerably 


84. 


4% 


2 


.8b 




workshops benefited 


b. 


Somewhat 


15.6 








classroom instruction at 


c. 


No observable 


0. 


0 








school 




effect 










2. 


AGE teachers shared their 


a. 


Frequently 


56. 


3% 


2 


.5b 




experiences with other 


b. 


Somewhat 


40. 


6 








teachers at school 


c. 


Not at all 


3. 


1 






3. 


Teachers and/or students 


a. 


Considerably 


78- 


1% 


2 


.8b 




benefited from partici- 


b. 


Somewhat 


21.9 








pating in AGE-sponsored 


c • 


No observable 


0- 


0 








external classroom special 




effect 












programs 


d. 


No special pro- 


0, 


0 












gram provided 










4. 


Teachers are interested in 


a. 


Considerable 


87. 


5% 


2 


.9b 




participating in future 




interest 












AGE training workshops 


b. 


Limited interest 


12. 


5 










c. 


No interest 


0. 


0 







= 32 

b"a" = 3, "b" = 2, "c" = 1, and "d" = 0 

• In general, evaluation of the program's impact by 
principals was very positive. 

• AGE training workshops benefited classroom instruction in 
over four-fifths of the cases. 
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the arts into their basic instructional programs. AGE teachers, 
however, should be further encouraged to share their learning 
experiences with other teachers in their schools. Efforts should 
also be made to have all teacher participants fill in the 
Workshop Evaluation Form in order to have a complete set of 
teacher responses. This will improve the evaluation of the 
project. 

Future evaluation objectives should include a quantifiable 
criterion for each measure of program success (for example, "each 
workshop will receive a mean rating of at least 20 from teachers. 
Teachers will receive a rating of at least 12 on the Teacher 
Survey, and principals will give the progrsun a rating of at least 
ten on the Principal's Survey). 
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APPENDIX A 

N«w Tork City Bo«r4 of C4uc«tioo 63418 

Offict of Sp«ci«l Projects 
Uvrcnct P. LArklii« Dir«ccor 



WORKSHOP EVALUATION PORM 



Dtar FartldpiDt: 



Your Input froB this coapltttd tvaluatloii fora vill astltt us 
to aodlfy tad laprov* futurt workshops. Thank you for your coopsrstlon« 

OIST&ICT 



DATE OP WORKSHOP. 
WORKSHOP TITU 



WORKSHOP LOCATION 



WORKSHOP LEADER 



WORKSHOP PARTICIPAKT KTLE: SUPERVISOR (ADMINISTRATOR) 

OTHER (SPECirr) TEACHER (GRADE LEVEL). 



DISTRICT STAFP 



PARAPRDPESSXONAL 



SCHOOL (OPTIONAL) 



NAME OF RESPONDENT (OPTIOMAL) 



STRONGLY 
AGREE 
5 



AGREE 



UNKCIOED 
3 



DISAGREE 

2 



STRONGLY 
DISAGREE 
1 



1. Tbs workshop vss sffsctlvs 
in prsssntlng ths sstsrlsl 
in s ssnnsr that vss ussful 
In ispxovlns cXsssrooa 
QStruetloo* 

Z» Ths BStsrlsls ussd vsrs 
rtltvsat sod spproprius to 
cht ro'lc, 

3. ihs goals and objsctlTSS 
of tha vorkshop vara achlavad. 

A. Workshop participant a 
i^uastiona vara aacourstsd 
and answartd profasalonsUy. 



5. Tha workshop daaonstra- 
:ion Xaadar vss knowladgaabla 
About tha progras? 



RECOWENDATIONS - GENERAL COMMENTS 



Firat: Plaasa llat froa your profasaloasl parcaptlooa and vorkshop azparianca tha 
thraa (3) soat significant poaltiva ftaturas» quslltias and/or highlights of tha 
vorkshop you just psrtldpatad In. Firat ittprasalons ara Uportant. Ona vord or a 
briaf santanca to dascrlba your praaant faaling is adaquatt. 

Posltiva Faaturaa of Workshop; 



1. 
2. 



0. Second: List an^ aagativ.a faallngs you hava about tha vorkshop axparianca. If noac, 
plaasa vrita nona. Taantc you. 
Nagativa Faallngs ; 

1 . 



ERLC 



.e >. .se cna back of this shaat for aoT aodicional racoaaatdations or ganaral co=:ienLs. 

Thank you for your resoonse. 
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1984-85 

ARTS IN GENERAL EDUCATION (AGE) 
TEACHERS' SURVEY 

TEACHER^S NAME GRADE LEVEL 

SCHOOL SPECIAL SUBJECT AREA 

^Cc Participation (circle one): Year 1 Year 2 Year 3 rtore 

1. The teacher's lesson plans indicate that she/he integrates arts activities 
into classroom instruction. 

a. never b. seldom c. sometimes d. frequently 

2. AGE teachers scheduled external arts activities for students. 

a. never b. seldom c. sometimes d. frequently 

3. The teacher provides instruction in basic art concepts. 

a. never b. seldom c. sometime d. frequently 

4. The students' behavior de.aonstrates enthusiastic response to arts activities* 
a. never b. seldom c. sometimes d. frequently 

5. The teacher encourages students* creative efforts. 

a. never b. seldom c. sometimes d. frequently 
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1984-85 

ARTS IN GENERAL EDUCATION (AGE) 
PRINCIPALS' SURVEY 



PRINCIPALS* NAME 



SCHOOL 



1. Skills learned by teachers In AGE training workshops benefited classroon 
Instruction In my school. 

a. considerably b. somewhat c, no observable effect 

2. AGE teachers shared their experiences with other teachers at my school, 
a. frequently b. somewhat c. not at all 

3. Teachers and/or students at my school benefited from participation In 
AGE-sponsored external classroom special programs. 

a. considerably b. somewhat c. no observable effect 
d. no external special program provided 

4. Teachers at my school are Interested In i^artlclpatlng In future AGE 
training workshops. 

a. yes» there l^ considerable Interest 

b. Interest Is limited to a few teachers 

c. no, teachers do not want to participate 

5. Cite at least one specific example of hov AGE participation contributed 
to your school program this year. 
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ENRICHMENT PROGRAM 1985-86 

School-Community Education Program 
Program Administrators Jack Isaacs 
Project Coordinator: Barbara Slatin 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

The Enrichment Program *.-9 provides staff development 
workshops to elementary and intermediate school teachers in 
Community School Districts (C.S.D.s) 11, 21, 24, 25, 29 and 29. 
The purpose is to motivate and train teachers for them o meet 
the needs of high-achieving pupils in kindergarten through grade 
nine. 

In 1985-86, about 178 teachers were selected by their prin- 
cipals to participate in the project. They attended five all-day 
workshops, focusing on such topics as assessment of the 
instructional needs of gifted students, Taylor's Multiple Talert 
Theory, and the Enrichment Renzulli Triad Model. The workshops 
were conducted by district staff and consultants in gifted 
education who alsc visited the classroom of each participant to 
pr /ide assist^ ^e in implementing project activities. 

The objective for 1985-86 was for participants to improve 
their knowledge of teaching techniques in the areas of instruc- 
tional management, reasoning skills, and curriculum enrichmenc as 
measured by their performance on a project-developed test. The 
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project received $108 thousand in funding from the New York State 

Legislature. 

EVALUATION METHODOLOGY 

Project impact was assessed by an analysis of teachers' 
scores on a project-developed test (see Appendix A), The 20-item 
test measures knowledge of Taylor's Multiple Talent Theory, forms 
of thinking, and strategies for gifted education. The test was 
administered on a pretest and posttest basis at the beginning and 
end of the program. 

FINDINGS 

Complete test scores were reported for 164 teachers from all 
districts except C.S.D. 29 (see Table 1). Overall, mean gain was 
22 percent. Pretest mean raw score for all districts was 14.3 
points (71.5 percent correct responses) and posttest mean raw 
score was 18.7 points (93.5 percent correct responses). Analysis 
of test scores by district shows that mean pretest raw scores 
ranged from 13 to 15.8 points (teachers in all districts cor- 
rectly answered above 65 percent test items) and posttest raw 
scores ranged from 17.7 to 19.7 points (teachers in all districts 
correctly answered above 88.5 percent test items). 

CO NCLUSIONS AND RECOMMENDATIONS 

The Enrichment Program K-9 was successful as it met its 
stated objective. Teachers in all districts improved their 
knowledge of teaching concepts and techniques but the average 
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TABLE 1 

Teachers Mean Raw Scores^ on a Program-D3ve loped Test, 

By District 
Enrichment Program K-9, 1985-86 



Pretest 



Posttest 



Raw Percent 



Raw 



Percent 



Gain 



District 


N 


Score 


Correct 


Score 


Correct 


Raw 


Percent 


11 


21 


13.0 


65.0% 


17.7 


88.5% 


4.7 


23.5% 


21 


18 


14.2 


71.0 


18.7 


93.5 


4.5 


22.5 


24 


70 


15.8 


79.0 


19.7 


98.5 


3.9 


19.5 


25 


22 


13.3 


66.5 


19.2 


96.0 


5.9 


29.5 


28 


33 


15.2 


76.0 


18.2 


91.0 


3.0 


15.0 


TOTAL 


164 


14.3 


71.5 


18.7 


93.5 


4.4 


22.0 


^Perfect 


Raw Score 


= 20. 













• Overall mean gain was 22 percent points. 
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improvement was low at 22 percent points. The reason for this 
was that pretest scores were too high to allow for further 
improvement (ceiling effect). Most teachers knew more than 65 
percent of test items at pretest, indicating that the test was 
too easy for them. Previous evaluation findings and 
recommendations have indicated that the test needs to be revised, 
eliminating those items that most teachers know before 
participating in the project. The Office of Educational 
Assessment offers the same recommendation made in previous years. 
In addition, future project objectives should include a 
quantifiable measure of achievement (for example, 75 percent of 
the teachers will achieve a gain of at least 30 percent) . 
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APPENDIX A 
ENRICHMENT PROGRAM K-9 



NAME_ DATE 



CLASS SCHOOL 



1) Calvin Taylor's approach to the teacher-learning process is called 
the; 

a) multiple talent approach 

b) content process approach 

c) product orientation method 



2) In view of what you know o: Taylor's Rationale, which of the 
statements listed below would best describe a talent implementation 
program in the classroom? 

a) separately from the acquisition of knowledge 

b) simultaneously with the acquisition of knowledge 

c) alternately with the acquisition 

3) Select the component (s) which are invorporated in the complex 
process of the Multiple Talent Approach to learning: 

a) cognitive 

b) affective 

c) neither of these 

d) both of these 

4) If you had a class from a low-socio economic background, what could 
you expect of them in talent development? Choose the statement you 
feel is most accurate. 

a) some would be talented in all areas 

b) given enough time 85% would show achievement in several areas 

c) 9 out of 10 employ at least one talent with above-average 
efficiency both for acquiring knowledge and for solving problems 

T F 

5) When we speak of "gifted" students we are referring to 

a very homogeneous group of individuals. 

6) The individual intelligence test is the only true 

indicator of giftedness. 

7) The gifted program should be separate and independent 

of other school programs. 

8) The gifted program should be concerned with providing 

learning opportunities and experiences that will make 

up for deficiencies inthe regular classroom. 
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EARLY CHILDHOOD ENRICHMENT PROGRAM 



1985-86 



T 



F 



9) It is really i.nportant for the gifted program to have 
a separate and unique identity in your school. 

10) Divergent thinking is a type of thinking where there 
io uj;ually one answer. 

11) Remembering and recognizing information is the 
student's main job. 

12) The studen't job is to know the best answer to each 
problem. 



CHOOSE THE APPROPRIATE COGNITIVE LEVEL IN BLOOM'S TAXONOMY 

A) Knowledge 

B) Comprehension 

C) Application 

D) Analysis 

E) Synthesis 

F) . Evaluation 

13) Activities calling for selection of appropriate methods and 
performance of operations required by problem situations 



14) Activities calling for development and application of a set of 
standards for judging worth 



15) Activities calling for the generation of new ideas and solutions 



16) Activities calling for the recognition of the structure of 
material, including the conditions that affect the way it fits 
together 

17) Explain, reword, recognize, and outline 



18) Create, develop, originate, devise 



19) Match, list, write, recite 



20) Apply, solve, employ, construct, use 
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ADVENTURES IN SCIENCE, 1985-86 

School-Community Education Program 
Program Coordinator: Jack Isaacs 
Project Director: Frank Quinones 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

In 1985-86, the Adventures in Science Project was redesigned 
to provide an effective science training program to teachers and 
supervisors in Community School District (C.S.D.) 8. Recognizing 
the need to improve science instruction and curricula, the 
project trained and assisted participants in the development and 
implementation of appropriate science lessons. 

About 56 elementary and intermediate school teachers in 
grades three through five and supervisors participated in the 
project. School principals selected teacher participants among 
volunteers who expressed the need for assistance in science 
instruction. The training program was carried out in in-school 
and after-school workshops which were conducted by C.S.D. 8 staff 
and consultants. Teachers received further assistance in the 
selection and use of classroom materials, organization of class- 
room science centers as well as through demonstration lessons. 

The project objective was for teacher participants to 
improve their ability to develop and implement appropriate 
science lessons from material suggested in the N.Y.C. Board of 
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Education's Minimum Teaching Essentials, as measured by their 
performance on a project-developed test. 

4 

The New York State Legislature provided $30 thousand in 
funding to purchase training supplies, support teachers' after- 
school activities, and cover consultants' fees. 

EVALUATION METHODOLOGY 

Project impact was assessed by an analysis of teachers' 
scores on a project-developed test (see Appendix A). This test, 
consisting of 20 multiple-choice items, measures basic knowledge 
on science concepts and experiments. It was administered as both 
a pretest and posttest at the beginning and end of the program. 

FINDINGS 

Pre- and posttest scores were submitted for 42 teachers. 
Overall, mean raw pretest score was 17.2 points (86 percent 
correct), mean raw posttest score was 18.3 points (91.5 percent 
correct), for a mean gain of 5.5 percent. Table 1 shows the 
performance of teachers according to frequency distribution of 
mean gains. About 40 percent of teachers did not achieve any 
gains, 55 percent achieved gains from one to three points and 
only 4.8 percent achieved gains of five or more points. 

CONCLUSIONS AND RECOMMENDATIONS 

The evaluation findings indicate that although teacher 
participants improved their overall performance on the project- 
developed test at posttest, the mean gain was too small to 

2 
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TABLE 1 



Performance of Teachers on the 

Project-Developed Test 
Adventures in Science, 1985-86 



N 


Percent 


Gain 


17 


40.5 


0 points 


12 


28.6 


1 point 


6 


14.3 


2 points 


2 


4.8 


3 points 


3 


7.1 


4 points 


2 


4.8 


5 or more points 



• Over two-fifths of the participants 
(43 percent) achieved gains of one or 
two points. 
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conclude that the project was successful. The test is obviously 
too easy since participants correctly answered an average of 86 
percent correct responses of test items at pretest which, in 
turn, did not allow for further iinproveinent at posttest (ceiling 
effect). This is demonstrated by the fact that about two-thirds 
of participants achieved gains of one point. In addition, the 
testing instrument measures conceptual and factual science 
knowledge but cannot measure the project objective for teachers 
to improve their ability to develop and implement appropriate 
science lessons. Project staff should revise the evaluation 
instrument, deleting those items teachers know before partici- 
pating in the program and adding items about how to incorporate 
their knowledge in the classroom. An in-classroom evaluation of 
teacher performance by school principals, however, would provide 
more adequate information. Finally, the objective should include 
specific quantitative criteria for successful program completion. 
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APPENDIX A 



508 1 JO iifl g :< •rw'''.. lopp^A^ 
CirMi;,t UMBRELLA aUREAU i9Hb-db 
COMMUNITY SCHOOL DISTRICT 8 



AOVPNTURES IN SCIENCE 



Adv.nl,^« 1"? "'^ ' * '""J' '^"^ "0* you rernoDbcr about the 

tt,Ti. J. '^"^'■•T- ^"^ *" t"* SP»« ""t " the letter that lol 

mn M ms";Jw'A?"lT "LEASE oM" 

1. A chemical ch«n9e is one that; 

a. can not go back to the starting raitenals 
0. can go back to the starting materials 

c. disappear^ 

d. will produce no changes at all 

2. If you want a liquid to evaporate, you might: 

4. freeze it 

b. born it 

c. mix it with another liquio 
• d. boil it 



on a w:ndow 



a- U will becQoe invisible 

b. it will get lighter 

d.' it will ^^^rllt ^^^'^ l^^?"!^ h« evaporated 

^^^^5^; i-t«aybe,.n to bubble. This might be 

a. evaporation 

b. a piysical change 

c. condensation 

d- a chemical change 

5. All of the following are needed for burning except : 

a. fuel 

b. oxygen 

c. heat source 

d. carbon dioxide 

0- The Solar System is made chiefly of: 

a. stars 

b. planets 

c. planets and astenods 

d. The sun, the planets and their jioons 

7. Tne moon can be sesn from the Earth oecause 

a. It emits light 

b. It IS a light color 

c. It reflects lignt fron the sun 

d. It IS a satellite 

8. Light travels at: 

a. 8 miles per minute 

b. 186.000 miles per second 

c. 100 million miles per day 
a 60 miles per hour 

' Uftude IS -nedSj'-ed in 



•J inches 

b decrees 

C 2»*^niS 

(3 -t 1 ; es 

Er|c 81 
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0. w..:an not ooserve stars during the day time because- 

I they reflect the sun's light 

b. they emit light only a: night 

c. of the sun's brightness 
«3. they more in space 

n. Green plants use light. i.ater and carbon dioxide to produce- 



A- bacteria 

b. food 

c. minerals 

d. fungus 



7. 



YOU can Observe your face in a mirror because t.he mirror: 

*• produces light 

b. breaks down the light rays 

c. reflects light 

d. absorbs light 

13. Electricity circulates best via: 

a. air 

b. wood 

c. paper 

d . wi re 

'4-. A dry cell produces electricity becau5e-of: 

4- wind energy 

b. chemical action 

c. water energy 

d. heat energy 



!5. -h,ch Of the following di,gra«s represents a complete crcuu? 



0 




© 




n"u'"i: irili:'^'" * -^h h,s f,st an. ann. he ,s causing the t.,g,ng 

«. longer /' 

b- shorter ^ 

c. related 

d. none of the above 

I^. When rays of light strike a rough surface, they are: 

a. reflecced in many directions 

b. reflected in one direction 

c. absorbed by the surface 

d. destroyed 

S . A mirror can be used for all of the following except: 

a. looking over a will 

b. seeing behind you 

c. looking around a corner 

d. seeing in the dark 

>. Patterns of ™any snail objects can be »acle by us,ng m,rrors ,n a 

a. radioscope 

b. kaleidoscope 

c. gravity scope 
telescope 

^ When the amount of light falhng un an object reduced 

•J the image becomes &rinr.ter 

b. tne image is curved 

c. the ifrage DecofT^e* 
d z^e image di<:,>{^^»j,< 
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EARLY CHILDHOOD LANGUAGE AND LITERACY, 1985-86 

School -Community Education Program 
Program Administrator: Jack Isaacs 
Project Coordinator: Miriam Sour 

Prepared by: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

The Early Childhc Language and Literacy Projec'« is 
desi ned to provide *:raining in communication arts to kindergar- 
ten teachers in Community School District (C.S.D.) 9. The 
purpose of the project is to teach participants the necer:^ary 
techniques and strategies to actively engage pupils in a struc- 
tured program in order co improve their listening, reading, and 
thinking skills. In 1986, the program was presented at the 
Interna clonal Reading Association Conference in Philadelphia and 
at the World Congress of Reading in London. 

Although in 1985-b6 the project originally intended to serve 
ten kindergarten teachers, it added an additional teacher in that 
grade and 12 first-grade teachers. This not only expanded the 
number of participants in the program hxiz also benefited lirst- 
year students who had been involved in the program the previous 
year. Participants were selected among volunteer teachers from 
P.S. 73 and P.S. 104 '/here low pupil achievement indicated a need 
to improve teaching techniques in communication arts. 



83 



63432 

The project objective was for teacher participants to 
achieve an increase in their ability to teach communication arts 
to kindergarten and first-grade pupils as measured by a program- 
developed survey, ''he inventory was administered at the begin- 
ning and the end of project activities. 

Staff members consisted of a project director and one 
teacher-trainer consultant who visited the schools and classrooms 
twice a week to provide project services. These included 
demonstration lessons, workshops, and articulation of program 
procedures. Teachers were shown how to organize their classrooms 
so that there were reading corners, listening centers, art areas, 
writing centers, and language development geurie areas. These 
areas could be used for whole group, small group and individual 
pupil activities. Specially designed materials such as language 
development games, big books, a library of books for individual 
selection and audio-cassettes for student practice in listening 
skills were usea in the classrooms. The New York State Legisla- 
ture, contributed $iy thousand to pay for the consultant's 
services and to purchase educational supplies. 

EVALUATION METHODOLOGY 

Project evaluation was based on analyses of teachers' 
holistic scores on a 16-item program-developed Early Childhood 
Languag'^ and Literacy Survsy designed to measure teachers' 
ability teach communication arts (see Appendix A). Teacher 
responses to survey questions were holistically scored by the 
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C.S.D. 9 project coordinator but the guidelines used for scoring 
responses were net submitted to the Office of Educational 
Assessment. 

FINDINGS 

Holistic pre- and post-program scores for 23 participants 
were submitted for evaluation tcgether with a written report, 
detailing project activities and accomplishments. Table 1 
presents teacher survey outcomes by grade. Overall, mean pre- 
program score was 3.3. points (20.7 percent) mean post-program 
score was 14.4 points (90 percent), for a mean gain of 11.1 
points or 65? 3 percent. First-grade teachers scored slightly 
lower than kindergarten teachers when surveyed before starting 
project activities but achieved a higher g^in by the end of the 
program. 

CONCLUSIONS AND REC0MI4ENDATI0MS 

The Early Childhood Language and Literacy Project was a 
successful program having a significant impact on teacher ability 
to teach communication arts. Teachers achieved an overall mean 
gain of 11.1 raw points (about 69 percent increase), a very large 
gain indeed. In spite of this improvement, it remains difficult 
to determine how teacher survey responses were scored since 
scoring guidelines were not provided. In the future, these 
should be submitted together with survey scores. In addition, 
project staff should expand the program's objective to include a 

3 
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TABLE 1 

Teachers' Mean Raw Scores on a Project-Developed Inventory^ 
Early Childhood Language and Literacy, 1985-86 



Pretest Posttest 



Grade 


N 


Raw 
Score 


Percent 
Correct 


Raw 
Score 


Percent 
Correct 


Gain 


% 


K 


11 


3.8 


23.8% 


14.5 


90.6% 


10.7 


66.8 


1 


12 


2.8 


17.5 


14.3 


89.4 


11.5 


71.9 


TOTAL 


23 


3.3 


20.7 


14.4 


90.0 


11.1 


69.3 


^Perfect 


Score 


= IC 













Teachers achieved an overall mean gain of 69 percent. 
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quantifiable measure of successful program completion. The 
following sentence, for instance, could be added to the objective 
"Teacher participants will increase their ability to teach 
communication arts by at least ten points." 
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APPENDIX A 



Early Childhood Language and Literacy Survey 



1. Children who are talking to each other are learning less than 
children who are working on an individual task. 

Always Usually Occasionally Never 



2. A teacher is telling a story to a group of tea pupils.. The 
teacher wants to determine v aether the pupils are actively 
involved in listening. The best way for a teacher to do this 



a. Notice if you can make direct eye contact with 
each pupil 

b. Observe that all ten pupils are sitting quietly and 
listening to you 

c Ask a question and call on a randomly selected 
student 

d. Ask a question and call on a volunteer student 

e. Randomly select a student to retell some part of 
the story 



3. What is the necessary prerequisite for beginning reading instruction 
with Kindergarten children? 

a. Knowing the aames Ox all the letters 

b. Knowing all the initial consonant sounds 

c. Knowi:»sc at least eight consonant sounds 

d. Knowing the naires of at least eight letters 
c. None of the abo e 
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4. What should be the teacher's most important goal in the teaching 
of writing in the Kindergarten year? 

a. To develop legible handwriting 

b. To develop basic encoding skills 

c. To develop interest in written communication 

d. To extend vocabulary 

5. What is the most important consideration for the Kindergarten 
teacher when arranging the seating in a classroom? 

a. Pupils can see th^ chalkboard 

b. Pupils can easily talk to each other 

c. Pupils and their activities are clearly visible to 
the teacher from the front of the room 

d. Pupils have designated seats they always use 

e. Pupils have space for easy movement in the room 

6. When is the best time to start teaching a child to read? 

a. At entry into first grade 

b. In the middle of the Kindergarten year 

c. When the pupil has mastered the necessary pre-reading 
skills 

d. When the pupil shows interest in printed language 

e* As soon as the pupil has reached the operational 
stage of cognitive development 

7. Kindergarten teachers are advised to have a variety of materials 
in their classrooms. Group games are included to promote 
particular skills. Select the most valuable aim for using these 
games . 

a. Vocabulary extension 

b. Small muscle development 

c. Visual discriminat i.on 

d. Eye/hand coordination 

e. Social development 
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8. At the beginning of the Kindergarten year, the teaching should 
concentrate on: 

a. Listening and speaking 

b. Listening, speaking and reading 

c. Listening, speaking, reading and writing 

d. Reading 

e. Reading and writing 

9. What should be the Kindergarten teacher's attitude toward 
children's reading errors? 

a. Ignore them, so as not to damage the child's confidence 

b. Correct errors on high-frequency words only 

c. Correct errors that interfere with understanding 

d. Correct all errors as they occur 

e. Correct all serious errors 

10. The major portion of an All Day Kindergarten class should be 
devoted lo: 

a. Work on assigned task 

b. Peer group interaction 

c. Individual pupil teaching 

d. Small group instruction 

e. Whole class lessons 

11. What is the most effective way of promoting spoken language in 
the Kindergarten classroom? 

a. Structured vocabuxary exercises 

b. Helping pupils to develop correct pronunciation 

c. Talking with small groups of children 

d. Pupils talking to each other 
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12. To teach reading in a Kindergarten class, the most important 
resource is: 

a. A pre-primer series 

Several different pre-prirrer series 

c. Trade books with controlled vocabulary 

d. Trade books without controlled vocabulary 

13. When children enter school they should be encouraged to express 
their ideas in writing: 

a. As soon as they enter school 

b. As socn as they can read some words 

c. As soon as they learn to make shapes 

d. As soon as they have learned to write some letters 

14. Every Kindergarten classroom should have a sufficient supply of 
books. The books should be: 

a. In the library area of the room 

b» In every area of the room 

c. In the students* desks or tables 

d. On shelves and distributed by the teacher 

15. If you had a choice of the approach to use for teaching reading 
to Kindergarten children, would you choose: 

a. Phonics approach 

b. Linguistic app'^oach 

c. Sight Word approach 

d. Language Experience approach 

e. Individualized Reading approach 

f. other - please specify 

Give reasons for your choice: 
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16. Teacher training is considered an important aspect in growth 
for all staff. Rank the following methods of training from 
1 (the most valuable) to 6 (the least valuable). 



Conferences 

Observations by your immediate supervisor 

Workshops 

Curriculum manuals 

In-Service Trainings 

Demonstrations lessons 

On-Site support 

College courses 



Name: 



District : 



Grade: 



Date: 
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DISCOVERING ABILITIES AND IMPROVING ACHIEVEMENT, 1985-86 

School-Community Education Program 
Program Administrator: Jack Isaacs 
Project Coordinator: C. Raseh Nagi 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

The Discovering Abilities and Improving Achievement Program 
is designed to train teachers in Community School District 
(C^S.D.) 22 to diagnose pupil abilities and prescribe appropriate 
educational activities. This enables teachers to work with both 
gifted and talented pupils as well as with those in need of 
remedial instruction. The project first implemented in 1985*86 ^ 
served 120 elementary school teachers who were selected by school 
principals among those willing to participate in the program. 

The project coordinator and consultants conducted all-day 
training workshops during September, 1985. The training design 
was based on the Structure of Intellect (SOI) model, developed by 
Dr. J. P. Guilford and enhanced by Dr. Mary Meeker, which focuses 
on the diagnosis of student abilities and the development of 
individual prescriptive learning activities. Teachers were 
trained tc develop students' cognitive skills, to differentiate 
the curriculum for potentially gifted pupils, and to implement a 
diagnostic/prescriptive classroom program. Project staff also 
assisted teachers with follow-up activities and classroom visits. 
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The project objective for 1985-86 was for participants tr 
demonstrate an ability to implement a diagnostic/prescriptive 
critical thinking program based on the SOI theory. Teacher 
performance was measured by a project-developed test. The New 
Yo^'k State Legislature contributed $9 thousand in funding to 
cover expenses for substitute teachers. 

EVALUATION METHODOLOGY AND FINDINGS 

The evaluation of the project was based on analysis of 
teacher performance on a project-developed test consisting of 16 
multiple-choice items (See Appendix A). Pretest and posttest 
mean raw scores were compared to determine achievement differ- 
ences • 

Complete test data were submitted for 113 teachers. Pretest 
mean score was 9.6 points (60 percent correct responses); 
posttest mean score was 11.9 points (74.4 percent correct), for a 
mean gain of 2.3 points or 14.4 percent. Table 1 presents the 
frequency distiibution of teachers' raw gains. About 43 percent 
of participants achieved gains from one to five points; about 26 
percent achieved gains larger than five and 30 percent did not 
achieve any gains. 

CONCLUSIONS AND RECOMMENDATIONS 

The evaluation findings indicate that teachers improved 
their performance on the project-developed test c\t posttest. 
Overall, mean gain was, however, modest (2.3 raw points) and only 
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TABLE 1 

Frequency Distribution of Teachers' Raw Gains 
on a Project-Developed Test^ 
Discovering Abilities and Improving Achievement, 1985-86 



Raw Gain 


N 


% 


0 


34 


30.1% 


1-5 


49 


43.4 


6-10 


28 


24.8 


+ 10 


2 


1.7 


TOTAL 


113 


100.0 



^Perfect Raw Score =16 



• About a third of teacher participants achieved gains, 
ranging from one to five raw points. 
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about 25 percent of participants achieved c» gain larger than 
five points. These findings indicate chat the testing instrument- 
might be too easy since participants correctly answered more than 
60 percent items at pretest. In addition, the posttest mean 
score was relatively low which could reflect a lack of correspon- 
dence between project activities and what the test actually 
measures r The test should be revised to eliminate t-ie items 
teachers know prior to participation in the program and include 
items that effectively reflect project activities. 

A further problem with the testing instrument is that it 
cannot measure the program objective or the teachers' " JDility to 
implement a diagnostic/prescriptive critical thinking program." 
Instead, the test measures factual knowledge about the operations 
of intellectual ability. Thus, it remains difficult to determine 
whether the project succeeded in meeting its objective in spite 
of the teachers overall improvement. In order to evaluate 
effectively teacher ability to implement a diagnostic/prescrip- 
tive critical thinking program in the classroom, pertinent 
questions should be added to the testing instrument, for 
instance, what skills should be taught, when, in what order, to 
which students, and so on. Another way of assessing teacher 
performance would require project staff to appraise teachers in 
in-classrooni situations according to a program-developed teacher 
evaluation form. Finally, project staff should establish a 
specific que?ntitative criterion for successful program comple- 
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tion. The objective could indicate, for instance, that 80 
percent of the participants will achieve a gain of at least 25 
percent. 
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APPENDIX A 

•DISCOVERING ABILITIES AND IMPROVING ACHIEVEMENT 

PRE -POST TEST 

What area of Intellectual Ability is assessed by these questions. 

1. Which sounds (shapes)are alike? Which ones can be put together? 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

2. Alphabetize these words. Put the numbers that are alike together. 

a) cognition b) memory c} evaluation d) convergent production 
e) divergent production 

3. What do these words mean? (vocabulary) 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

4. Do you remember which figure goes with thif one? 

a) cognition b) memory c) evaluation d) convergent production 
e^ diver<»ent production 

5. What card did I just show you? (playing cards) 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

6. In the story ws read, who was the main character? h'hat did he do? 
hTio was his friend? Where was he from? Etc... 

a) cognition b) memor>' c) evaluation d) convergent production 
e) divergent production 

7. Find two objects that are related to each other. Why are they re 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

8. Which of these words are related to each other because of the way 
they are spelled? or sound? 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

9) Which words or ideas go together? Why? 

a) cognition b)memory c) evaluation d) convergent production 
e) divergent production 

10) Put these pictures in order that they should go in. 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent productioi 

11) What is 1 * 3? (5 - 4? 8 X 2? etc.) What is a four letter wcrd 
that starts with M and ends with E? 
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a) cognition b) memory c) evaluation dj convergent production 
e) divergent production 

12. If you did this particular task, or used this tool, what would 
your occupation be? 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

13. Make something out of this clay, paper, tile, etc. 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

14. Take all these noses and ears and things and see if you can make 
new faces. 

a) cognition b) memor>^ c) evaluation d) convergent production 
e) divergent proauction 

15. Make a new word with the ending letter of this word. Rewrite 
this song or rhyme. 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

16. Can you write a poem: 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 
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SUM ir ONE, 1985-86 



School-Community Education Prograun 
Program Administrator: Jack Isaacs 
Project Coordinator: Barbara Herman 



Prepared By: 
Office of Educational Assessment 
New York City Public Schools 



PROJECT DESCRIPTION 

The Sum in One Program provides staff development services 
to teachers in order to improve music instruction in elementary 
schools. By supplementing and upgrading music instructional 
techniques, the program seeks to involve a cadre of teachers who 
will, in turn, assist in the development of other teachers. In 
1985-86, 80 teachers from Community School District (C.S.D.) 1 
participated in the program. 

Participants were selected among those teachers who 
expressed a need for additional training in music instructional 
techniques and were interested in the project. School principals 
re ommended them for program participation and they were finally 
selected by the program coordinator. Instructional activities 
involved a series of three workshops, conducted by the project 
coordinator, which were held during regular school hours. The 
workshop series were the following: 

• Music in-Service Workshop 

• Staff Recorder Workshops 

• Staff Piano Workshops 

Instruction focused on techniques to motivate children to make 
music, organize the room for music lessons, ceach basic music 
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skills r the soprano recorder^ and the piano. Project staff 
consisted of one teacher-coordinator. The New York State 
Legislature provided $17 thousand in funding to purchase instruc- 
tional supplies and equiimient. 

The program objective for 1985-86 was for 80 percent of 
teacher participants to increase their ability to extend and 
enrich their students' music experiences and activities. Teacher 
performance was measured by three forms of a program-developed 
test. 

EVALUATION METHODOLOGY 

Evaluation activities focused on analyses of teacher 
performance on three forms of a program-dpveloped test (see 
Appendix A). A different form was administered for each workshop 
series. Perfect score for each test form was 100 points. 
Teachers took the test at the beginning and at the end of project 
activities. 

FINDINGS 

Complete test scores were submitted for 30 teachers who 
attended different workshops. Seventeen teachers attended the 
Staff Piano Workshop (SPW)^ eight participated in the Music in- 
Service Workshop (MSW), and five teachers attended the Staff 
Recorder Workshop (SRW). All participants (100 percent) improved 
their performance at posttest. Table 1 shows evaluation findings 
by workshop. Mean pretest raw scores ranged from 6 to 28.6 
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TABLE 1 

Teachers' Mean Raw Scores^ on a Program-Developed 

Test, by Workshop 
Sum In Oner 1985-86 



Workshop 


N 


Pretest 


Posttest 


Gain 


SPW^ 


17 


19.5 


83.4 


63.9 


MSWC 


8 


28.6 


92.9 


64.3 


SR}^ 


5 


6.0 


87.2 


81.2 


TOTAL 


30 









^Perfect raw score on each workshop test = 100. 
^Staff Piano Workshop. 
^Music In-Service Workshop. 
^Staff Recorder Workshop. 



• Teacher participants achieved mean gains ranging from 
about 64 to 81 points. 
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points, mean posttest raw scores ranged from 83.4 to 92.9 points, 
for gains ranging from 63.9 to 81.2 points. 

CONCLUSIONS AND RECOMMENDATIONS 

The Slim in One project was highly successful. Participants 
(100 r-rcent) met the project objective and achieved considerable 
mean gains in all three workshop test forms. A comparison of 
pretest scores, which were relatively low, and posttest scores 
indicates that the project had a remarkable impact on teachers. 
In the future, however, project staff should establish specific 
quantitative criteria for successful program completion. A 
sentence stating, f.->r example, "80 percent of teacher partici- 
pants will show an increase of at least 50 percent," should be 
added to the program objective. 
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SU-i m 0«: Supplonantiiig and Upgrading Music in Di:;trict Oths 
PRE/rC5T r/.UUATiai: Piano Staff VJorkshops 
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1 2.3.4.5. Ust 5 ways in which a tcadier's piano skills can relate to the 
sluicnt's'acvclopjcnt aid the curriculum. 



6.7.8.9.10.11.12.13.14 Write the follcving scales in both the Ucble and bass clefs, 
juxl name each note. 




F Major 




15 IG.17.10 W.ile a 4 m3asure rhvU<n. dr^sir.g fro.. Lhc foUc-.v.r,^ -V^x-ls: 

^ i- -J d cJ. > J3 X ^ cJ 



IS. 20.21. 22. 23. 24. 25. 2G. Nai.c all Oxi i*otcc iii "/-V.^^rica. 



My coun - try. 
Our fa - ihcTs' 



Mis of Ihcc, 
God to Tbco, 



SwCCl land of 
Au • thor of 



lib - cr - ly, 
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hVsLch Lhc words and svnilols: 
27. eighth note 



(Write the appropriate nunLer in the brackets.) 



28. oottod half note 

29. treble c3ef sign 

30. bass clef sign 

31. crescendo 

32. diminucncto 

33. soft 

34. loud 
25. repeat 
36 wliolc note 

Play . song of your am choice that you have learned in this course. 

37.38.r.tes 39.40.rhyU^ 41.42.tc^ 43.44.touch 45.46.phrasing 

Sight read the following 2 measure j^asc. 

47. Clap tlx; rliyUm. 

48. Say the note names. 

49. Play the right hand. 

50. Play the left hand. 
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FR£/rG£7 r^T^LUATiaJ: Staff Pecorder Workshops 

1.2.3.4.'=. List five wavs in which the study of the recorder will relate to the 
develcprlnt of skills recessary for the sd-jxl curriculun. 

* i 



6.7.8. vrnat is the articulaticn: for the hr.ch register? 

for the middle register? 
for the iCT-/ register? _ 



r 

.10. vrr^t axe the irajor differer.ces bef.ieen Gecnan fir.cerir.g ar.d Baroque or English 
fincerL-.c? , ' ^ 

. U.12.13.14.15.16.17.18.19.20.21.22.23.24-.25._I;lar3^^^^ for the 



C D . e: 



OO 

o 
o 

o 
o 



oo 
o 
o 



o 



o o 
o 



oo 
o 
o 

o 
o 

Do 



oo 
o 
o 

o 
o 



OD 


OO 


00 


o 


o 


o 


O 


o 


o 


O 


o 


o 


O 


o 


o 


0 


0 





oo 
o 
o 

o 
o 

® 



°8 

-I 

o 

o- 



--t — 


D 


e 

1 


F 




Is 

3 O 


goo. 


000 


gooi 


°8 

O 


) O 
) O 

6 ^ 


6 
o 


o 

o 


O 
O 


o 
o 






© 


® 


® 



26 '7.28.29. V.-rite a 4 r^as-urc ^^^e ch^sir.g frcn th.e follc.L.g rhythr. s^.ricls: 
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Si5Si^35r36.37., tone 38., articulation phrasiiig etc., 39.40. 

The following song has been chosen for you to sightread on the recorder: 
41.42. Conduct the beats. 
43.44. Speak the rhythm. 
45.46. Say the notes. 
47.48.49.50. Play on the recorder. 
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tPTJlCST EVPllIATICN: -Music In-Seivice Ccurse D14-1.F84 

Match each of th.e follwing concepts with its roit aporcpriate source. Use this 
C03 to shior-^; year an?*i£r: D=Dalcr02e; K=Kodaly; lOOrff; S=Su2uki. 

1- F^^son sto'jld be able to leam a rrjsical instrunent as naturally as a 

STiall cTiild learns his moLher tcngve. 

2' ^Features prcceediiig frcn speech to instruicntal inusic. 

3- ^Prcnotes fluency in reading end writing music. 

^" ^Ceyelcps concepts in a logical sequer.ce: prepared, made conscious, practised! 

written. 

5« Teaches piano inprovisation as a final goal. 

C. Develcps inner hearing. 

"^^ special instrxxi^ntarium is the prrr^ary means for r-jsical cevelcp-erit. 

^' ^Sir.cL-.g is used as the prrr.jry instrurent in general r:usic c-ducation. 

9, Copy Lhe v.xDrds music belcv. 



rr 




Here Kc go round the mul-bcr-r)* bush. Tht tnul-bcr-ry bujii, the mul.ber-r)'bush; 



11, I?., 13. The follcv/ing sor.g has four phoreses. V?rite lo.:er c^a letters (a, b elc.) 
in ti\e circles to eIto^- e.e TliP^SZ FOFM. The first phrase (a) is irdicc.t-3:d for ycu 




a 



1^ 
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• • writ.' th« notes tor the foUeuing scilts *r.4 indicate t: ■ rsiJCoJ reeoriir fir.jetings. 
14.15. C .Major sale 



o 
o 



o 
o 




o 



g 
o 



o 



o 



bo 

g 

o 
o 




5 
o 



8 



C 

o 

I 



o 
o 



g 
o 



Co 

0 ' e) 



O 

O 
O 



O 



POOO'CQ 
COO 
COO 



O 

o 
0 



d 
0 



c 
c 

# 



c 

c 

f7 



CO 



16.17. C .VJjor 5c«'? 



1 



oo 
o 

"O 

o 



P8 

g 
o 

6^ 



oooo 



o 
g 

o 
o 



o 
g 

o 



°8°8 



O 

O 



g 

o 
o 



g 

o 
o 



o 
0^' 



8 
8 



00 oo 
o 

O! 



g 
o 



.g 

o 
o 

J* 



ogc^ogog 
o 



o 
o 



c 
c 



oo'oc 


o 


c 


o 


c 


o 


c 


c 


c 


i 


! 
1 



15. 19. F yajor Scale 
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• U3iiwj Uio classification syclau of iitisic instruicias devised by Cuit Cadis, 
idcjiUfy Uio iir.UixiciiUl faiidly to wliicli cadi of Uic lollcwiny iiiuLruiciits 
U?long5. Use Uie folla^ii/j code: I»ldio[Jonc; M«ttoiijraiioi)liCMic; A-^Acroijlioiic; 
CnllDidoi'liuiiO; l>L'lccU:oiiliuJ)C 

20 ^cacio 22 i^iano 24 ^clariiict 

21 recorder 23 xylriliojic 25 ^liand drun 



26. DcscriJjc a directed liotciujio activity for your otuJaitJ prior U *':cir u:.e of 
xhyLiin iiistruiCJitu 

27. Dcscrilxj a» ann oioial you would develop wiUi your class io prciia^e fur 
coikluctiiiri a rhyUmi bond. 

List five ways in wliicli music study will relate to Uie develofciiciit of s}dlis 
necessary for Uic scliool cuiriculuu. 



29. 
30. 
31. 



32. 



33. Use your luiaiaac instxuicat or one of tlic clasuroan iiistruncnts to play one of 
X})C following rliyLi&iis: 



i u ni 



34 30. 30. 37. Write a rour iicasuro plirasqcloosiiig Iran Ucfolla.;im 

ciiiijols. YOU may play it on your rliyUm uistnitciit. H \ f \ J J e*4 



nn 1 1 



n I n I 



n nil 



30 39. 40. 41. a«»=o one ol Uie pieces you have learned to play on the soprao 
reorder during Uiis course and pcrfonn it lor Uc cxanuncr. 42. 43. 44. 4^. 

•ilc lollwiir, =ono la= Ixcn d»=cn for you to siyl.Lsuio aixl play on U.c rocoiucr: 

40. Conduct Uio beats and 47. speak Uio rhyUin. 
40. Sing ill solfcge wiUi 49. liand signals. 
00. Play on Uio recorder. 



M Horses 




M hor-jes bell horocs, whaiSOiaimuf da^? 
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MASTERY LEARNING PROGRAM, 1985-86 

School-Community Education Program 
Program Administrator: Jack Isaacs 
Project Coordinator: Louis Leonini 

Prepared By: 
Office of Educational Assessment 
New York City Public Schools 

PROJECT DESCRIPTION 

This project is designed to train teachers and supervisors 
in Mastery Learning (M.L. ) methodology which is extended in 
workshops and through in-class support. M.L. is a teaching 
strategy to improve student learning skills and attitudes towards 
the school. Previous experience indicates that students 
receiving instruction based on M.L. techniques improve their 
grades and rate of attendance. 

In 1985-86, the project served about 146 elementary and 
intermediate school teachers and supervisors from 20 schools in 
Community School Districts (C.S.D.s) 9, 20 and 31. School 
principals selected participants from a list of volunteers. The 
stated project objective was to introduce Mastery Learning 
techniques and procedures to teachers and supervisors in three 
New York City school districts. This was measured by a project- 
developed test. 

One teacher-trainer conducted the workshops and supported 
teachers by offering in-class training assistance. Instructional 
activities focused on M.L. theory and practice, and on the 
development of M.L. tables of specifications, formative tests, 
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and correctives. The New York Legislature contributed $32 
thousand to cover salary expenses. 



EVALUATION METHODOLOGY 

Evaluation activities focused on the analysis of partici- 
pants' scores on a project-developed test especially designed to 
measure knowledge of M.L. principles and teaching techniques. 
The multiple-choice test consists of two parts. The first one, 
administered as a pretest, has ten questions formulated to 
evaluate basic kuo^lodge of M.L. methodology. The second part, 
administered as a posttest, consists of a different set of more 
specific questions (see Appendix A). 



Complete test scores were submitted for 89 participants (See 
Table 1). Overall, mean pretest raw score was 5*5 points (55 
percent correct responses) and a mean raw posttest score of 9 
points (90 percent correct). The average gain was 3.5 raw score 
points or 35 percent. 

CONCLUSIONS AND RECOMMENDATIONS 

The evaluation findings indicate that the Mastery Learning 
Program was successful. Participants improved their knowledge of 
the M.L. curriculum. Yet, several problems make it difficult to 
adequately assess project impact. First, the posttest contains 
the same or similar questions as the pretest and since partici- 
pants correctly answered more than half of test items on the 



FINDINGS 
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TABLE 1 



Participants' Mean Raw Scores on a Project-Developed Test* 
Mastery Learning Program, 1985-86 



Mean Percent 

Raw Score Correct 

Pretest*' 5.5 55.0% 

Posttest^ 9.0 90.0 

Gain 3.5 35.0 



^Perfect Raw Score = 10 
^ N = 89 



Participants achieved a mean gain of 3.5 raw score 
points. 
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pretest, they could not possibly make larger gains. Secondly, 
the project did not clearly specify an objective besides intro* 
ducing M.L. techniques and procedures to participants. In the 
future, project staff could consider revising the test, elimi - 
nating those items which most participants know at pretest and, 
preferably, using one test to administer both as a pretest and 
posttest. The project objective needs to be clearly defined and 
should include a quantitative criterion for project success. 
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Pretest 

Directions: Please place a check nexc to the answer you fe^l is the r.cst 
a?src?riate for each of the following tar. questions. ' 

1) The theory ur^erlyinq Mastery Learning was primarily developed by: 

a) Jases Colesian 
b) Jaa:ea P. Keller 

c) Senjarain Sloosi 
d) Richard Wolf 

2) Hastery Learning is a teaching/learning strategy. It is also 

a) A discipline-inter-related extension exercise 

b) An instructional philosophy 

c) A direct Instruction approach 

d) An individually-paced approach to teaching 

3) Master/ Learning has had the following affective results on th-s stu-lents 

a) It trains students to be . ne good test tu-ier^ 

fc) 2c teaches students how to learn 

c) Ic sahcs students feel that their teocher is their friend 

d) Iz cakes students learn faster 

4) In Mastery Learning the "correctives- can be done by 

a) The student hiaself ' ' ' ^. . 

b) The teacher 

c) The students and the peer tutors 

d) All of the above 

5) In Mastery Learning the term "correctives- applies to 

a) Varied test construction, extension activities 

b) EnrichiP.ent activities, learning/teaching styles 
c) Alternative texts, re-teaching, tutorir.c 

d) Teacher praise, public recognition, crises 

6) A "forsative test" in riastery Learning is helpful co both teacher and s'lxidenz 
in that it 

■ a) Provides inforaation to both teacher and student regarding 

the type of enrichment activities required 
b) Provides a psychological framework for teaching and learning 
to take place 

c) Provides feedback to both the toacher and students^ about the 

learning that has occurred up to thac point 
d) Mone of the above 

7) A "Table of Specifications" might also be called 

____ a) A lesson plan 

b) A Table of Contents 

c) An individual pupil profile sheet 
d) A Unit Plan 
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Paqe 2 - Questlcnnaire I 



8) Th« highest level Oi cognitive develo.oitient on the Table of Speci 

a) Analysis 

b) Evaluation 

c) Synthesis 

d) Translation 

9) The proeer sequence of events in the Mastery Learning apporach is as foil 

«) Formative Test A, Correctives, Enrichaent and Extension, 

Pozaative Test B 
b) Formative Test, Extension, Correctives, Suswiative Test 
• c)SuamAtive Test, Correctives, Formative Test A, Extens-on 
Activities, Formative Test B . ^ . ' 

d)Suipmative Test, Enrichment and Extension, Formative iest A, 

Correctives, Formative Test B 

10.' Which of the foUowing sample phrases or questions is an exacole of the 
translation level of cognitive development? 

aircan you develop a new way" 

~~~~ b) "Which one cones first?" 

'~~ c)"'^.iich do you consider of greater importance?" 
d) "Explain in your own words " 
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MAST£RY LEAPNING - QUESTIC^T^IAIHi: II 

Post test 

Directions: Please place a checR fiext to the answer you feel is the rest 
appropriate for each of the following ten questions 

1) Which of the following is a higher level of cognitive development? 

a) Tcra 

b) Principle 

c) Translation 
d) Process 



2) Mastery Learning is an instructional philosophy. It is also 

a) An individually paced approach to teaching 

b) A discipline-inter-related extension exercise 
r c) A teaching/learning strategy 
d) An instructional c\are 

3) Mastery Learning has had affective as well as instructional results. Cne 
affective result upon the student has been 

a) A stepped up student pace of learning 

b) A true friendship between teacher and studert 

c) A greater awareness in the art of test taking 
d) Instruction on how to learn 

4) Tho Mastery Leaminc; theory was prirriarily developed in the 1950 's by 

a) Ral h W. Tyler 

b) Peter Aisasian 

c) Richard Wolf 

d) Benjanin Bloom 

5) In Mastery Learning -correctives" can be done by 

a) The student himself 

b) The teacher designated 

c) The students, and teacher assistants 

d) All of the above 

6) In Mastery Learning the tern "correctives" can be appUed to 

a) Enrichment activities, learning/teaching styles 

b) Different types of tests, extension activities 

c) Peer tutoring^ work sheets, re-teaching 

d) Prizes r games ^ teacher and public recognition 

7) The eloent in Mastery Learning which pWdes feedback to both the teacher 
and students about the learning that has occurred up to a certain point is 

a) The Formative Test 

b) The Psychological framework Analysis Test 

c) The Enrichment/Extension Quiz 

d) Kone of the above 
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Pa9€ 2 - Questionnaire II 

8} The proper Mastery Learning sequence is as follows 

a) Summative Test, Enrichment and Extension Forrative Test A, 
Correctives, Fonaative Test B 

b) Suwaative Test, Correctives, Fomative Test A, Extension 
Activities, Fonaative Test B 

c) Formative Test, Extension, Correctives, Sissative Test 
d) Formative Test A, Correctives, Enrichrjsnt and Extension, 

Formative Teet B 

9) In Mastery Learning a "Unit Plan- might also be called 
a) A Table of Contents 

b) A Proftla Sheet 

c) A Table of Specifications 
• d) An instructional analysis sheet 

Yo) Which of. the following sample phrases or questions _isL.an. exjspl^ pf.. the. 
* Evaluation level of cognitive developcentrr: • - 

a) "Can you develop a new way? 

J b) "Which one comes first?" 

c) "Which do you consider of greater importance?" 

d) "Explain in your own words " 
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